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FOREWORD 


This  research  and  development  is  being  conducted  in  response  to  Navy  Decision 
Coordinating  Paper  (NDCP)  W0784-PN,  Simulated  Avionics  Maintenance  Trainer  (SAMT) 
Work  Unit  OWR01029,  under  the  sponsorship  of  the  Naval  Air  Systems  Command  (NAVAIR 
413).  The  objectives  of  the  NDCP  are  (1)  to  conduct  engineering  developments  for  a 
family  of  SAMTs,  (2)  to  develop  SAMT  trial  procurement  specifications  (organizational- 
and  intermediate-maintenance  levels),  (3)  to  use  the  trial  procurement  specifications  to 
procure  the  prototype  trainers,  (4)  to  evaluate  the  effectiveness  of  the  trainer,  and  (5)  to 
refine  the  procurement  specifications  based  on  an  evaluation  of  the  procurement  process 
and  the  trainer  effectiveness. 

The  objective  of  Work  Unit  OWR01029  is  to  develop  a  generic  specification  for  a 
mechanical  and  electromechanical  organizational-level  SAMT.  This  report  covers  the 
analysis  of  two  existing  generic  specifications  that  have  been  developed  for  the  Naval  Air 
Development  Center  and  the  Air  Force  Human  Resources  Laboratory  (AFHRL) 
respectively.  Results  of  the  analysis,  which  was  performed  by  Advanced  Technology,  Inc., 
San  Diego,  under  contract  N00244-79-C-1760,  will  be  used  by  NAVPERSRANDCEN  in 
developing  the  SAMT  generic  specification.  Dr.  M.  Flaningam  was  the  contracting 
officer's  technical  representative  for  this  effort. 

Appreciation  is  expressed  for  the  assistance  in  preparing  this  report  provided  by 
personnel  of  the  Technical  Training  Division,  AFHRL.  Particular  appreciation  is 
expressed  for  the  support  provided  by  Dr.  E.  Smith,  who  is  overseeing  the  Air  Force  study 
at  AFHRL. 


JAMES  F.  KELLY,  JR. 
Commanding  Officer 


JAMES  J.  REGAN 
Technical  Director 


SUMMARY 


Problem 


The  increasing  sophistication  of  aviation  equipment  imposes  more  sophisticated 
maintenance  procedures  and  increases  the  difficulty  of  providing  trained  maintenance 
personnel.  To  facilitate  and  improve  training,  the  Chief  of  Naval  Operations  has  selected 
the  Simulated  Avionics  Maintenance  Trainer  (SAMT)  for  procurement  and  use  at  training 
schools.  The  Navy  Personnel  Research  and  Development  Center  (NAVPERSRANDCEN) 
has  been  tasked  to  develop  a  mechanical  and  electromechanical  organizational-level 
SAMT  generic  specification  defining  the  system  evolution  from  concept,  through  develop¬ 
ment,  to  procurement. 

Objective 

The  objective  of  this  effort  was  to  analyze  two  existing  generic  specifications 
developed  for  the  Naval  Air  Development  Center  (NADC)  and  the  Air  Force  Human 
Resources  Laboratory  (AFHRL)  respectively.  Information  obtained  will  be  used  in 
developing  the  SAMT  generic  specifications. 

Approach 

The  Navy  and  Air  Force  studies  were  examined  as  to  their  format  and  ease  of 
comprehension,  completeness,  and  interrelation  among  document  sets.  The  processes 
described  by  these  studies  were  then  compared  as  to  their  assumptions,  ability  to  meet 
established  goals  and  objectives,  applications,  scope,  and  utility.  Finally,  the  two 
processes  were  compared  in  terms  of  three  procedure  areas: 

1.  Determining  training  requirements. 

2.  Selecting  media  to  meet  training  requirements. 

3.  Producing  a  procurement  specification  for  training  equipment  acquisition. 

Results  and  Conclusions 

1.  The  Air  Force's  procedural  requirements  document  covers  the  full  development 
cycles  of  a  maintenance-training  simulator  from  course  concept  to  final  acceptance. 

2.  Although  the  Air  Force  process  was  developed  for  the  Air  Force  environment  and 
was  tailored  to  specific  Air  Force  activities,  many  of  the  conceptual  problems  are 
common  to  the  Navy  and  Air  Force,  and  the  concepts,  procedures,  and  specifications 
appear  to  be  applicable  to  Navy  use.  The  process  of  developing  training  and  training- 
equipment  requirements  for  new  or  existing  systems  can  be  used  to  develop  any  type  of 
maintenance-trainer  simulator. 

3.  The  Navy  study  provides  extensive  information  on  the  procurement  process, 
including  data  on  government  requirements  for  simulator  design,  procurement  practices, 
sample  letters,  and  Request  for  Proposals. 

4.  The  Navy  study  provides  many  aids  for  use  by  the  government  for  simulator 
procurement.  These  aids  are  well  written  and  provide  much  information  unknown  to  some 
government  personnel. 


Recommendations 


1.  The  Air  Force's  training-requirement  determination  process  (with  media  selec¬ 
tion)  should  be  used  to  assist  in  the  definition  of  the  general  functional  characteristics  of 
a  maintenance-training  simulator. 

2.  The  Navy's  procurement  sample  specification,  with  supporting  data,  should  be 
used  to  assist  in  the  development  of  the  procurement  section  of  the  generic  specification. 

The  Navy  and  Air  Force  study  documents  cannot  be  used  verbatim  and  it  will  be 
difficult  to  transfer  the  information  from  its  original  application  to  the  SAMT  require¬ 
ments.  Potential  difficulties  are  outlined  in  this  report. 
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INTRODUCTION 


Problem 


The  increasing  sophistication  of  aviation  equipment  requires  more  sophisticated 
maintenance  procedures,  which,  in  turn,  impose  more  complex  training  for  maintenance 
personnel.  The  training  problem  is  exacerbated  by  the  lower  level  of  personnel  entering 
training  and  by  the  high  turnover  of  trained  personnel.  To  ensure  the  state  of  readiness 
that  must  be  maintained  by  our  naval  forces,  it  is  imperative  that  more  effective  training 
methods  for  maintenance  personnel  be  developed  and  at  a  lower  cost  per  student. 

One  method  of  improving  maintenance  training  and  reducing  training  costs  that  holds 
promise  involves  simulation  techniques  inserted  in  the  instruction  curricula.  Some  of  the 
potential  gains  from  these  techniques  are: 

1.  Reduced  cost  of  maintenance  training. 

2.  Increased  training-equipment  reliability. 

3.  Increased  safety  for  student  and  instructor  during  hazardous  activities. 

4.  More  effective  instruction  from  training  equipment. 

5.  Better  instruction  in  troubleshooting  by  inserting  malfunctions  under  instructor 
control  that  would  be  critical  and  uncommon  during  shipboard  operating  condi¬ 
tions. 

While  these  potential  benefits  of  simulation  techniques  are  fully  recognized  and  are 
being  applied,  they  have  not  yet  taken  effect,  and  to  date  little  improvement  in 
maintenance  training  has  been  realized.  To  accelerate  the  simulation  program,  at  the 
direction  of  CNO,  Navy  Decision  Coordinating  Paper  for  Simulated  Avionics  Maintenance 
Trainer  was  issued  12  September  1977  to  coordinate  the  development,  procurement,  and 
implementation  of  a  simulation  trainer  concept- -the  Simulated  Avionics  Maintenance 
Trainer  (SAMT).  The  SAMT  family  of  trainers  covers  maintenance  training  on  mechan¬ 
ical,  electromechanical,  and  electronic  systems  at  both  organizational  and  intermediate 
repair  levels.  The  Navy  Personnel  Research  and  Development  Center  (NAVPERSRAND- 
CEN)  was  tasked  with  the  development  of  a  generic  specification  for  a  electromechanical 
and  mechanical  SAMT  at  the  organizational  level  (O-level). 

Objective 

The  objective  of  the  effort  described  here  was  to  analyze  the  documents  supporting 
two  existing  generic  specifications  developed  for  the  Navy  and  the  Air  Force  and  to 
extract  information  of  use  to  the  NAVPERSRANDCEN  task. 


APPROACH 


The  Navy  and  Air  Force  studies  were  examined  as  to  their  format  and  ease  of 
comprehension,  completeness,  and  interrelation  among  document  sets.  The  processes 
described  by  these  studies  were  then  compared  as  to  their  assumptions,  ability  to  meet 
established  goals  and  objectives,  applications,  scope,  and  utility.  Finally,  the  two 
processes  were  compared  in  terms  of  three  procedure  areas: 

1.  Determining  training  requirements. 

2.  Selecting  media  to  meet  training  requirements. 

3.  Producing  a  procurement  specification  for  training  equipment  acquisition. 
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THE  NAVY  STUDY 


The  Navy  study  consists  of  the  following  three  documents,  prepared  for  NADC  by 
Applied  Science  Associates  ( *\SA),  Inc.  under  contract  N62269-77-C-0304: 

1.  Development  Guidelines  for  Specifying  Functional  Characteristics  of  Mainte¬ 
nance  Training,  February  1978. 

2.  Procurement  of  Simulation  Systems  for  Training  in  Organizational-Level  Elec¬ 
tronics  Maintenance  at  Naval  Aviation  Maintenance  Training  Detachments:  Phase 

I- -Task  Analysis  and  Simulator  Configuration. 

3.  Procurement  of  Simulation  Systems  for  Training  in  Organizational-Level  Elec¬ 
tronics  Maintenance  at  Naval  Aviation  Maintenance  Training  Detachments;  Phase 

II-  -Specification  for  Procurement  of  a  Simulator  System. 

Hereafter,  these  documents  shall  be  referred  to  as  Development  Guidelines  (N),  Phase  I 
Document  (N),  and  Phase  II  Document  (N)  respectively. 

Development  Guidelines  (N)  details  a  process  for  the  development  of  maintenance 
training  simulators.  Phase  I  and  Phase  II  Document  detail  a  process  that  was  derived  from 
the  Development  Guidelines  (N)  process.  The  purposes  of  these  documents  were  to: 

1.  Produce  an  actual  procurement  specification  for  a  simulator  to  be  used  with  a 
Naval  Aviation  Maintenance  Training  Detachment  (NAMTRADET)  course  in 
organizational  maintenance  of  the  A-6E  TRAM/DRS. 

2.  Note  the  practical  concerns  of  both  the  development  contractor  and  the 
government  program  manager  and  improve  the  development  process  by  specifying  for 
future  program  managers  (a)  a  generic  development  process,  (b)  information  requirements, 
(c)  possible  information  sources,  (d)  review  points,  and  (e)  critical  features  of  program 
management. 

The  remainder  of  this  section  provides  a  synopsis  of  the  three  Navy  documents. 
Headings  are  the  actual  headings  from  the  Tables  of  Contents  (see  Appendix  A)  of  the 
documents;  all  statements  are  abridgements  of  the  document  text  without  comment  or 
amplification. 

Development  Guidelines  (N) 

Section  1.  Introduction 


Development  Guidelines  (N)  covers  the  following  areas: 

1.  A  process  for  functional  simulator  design. 

2.  Rules  for  and  responsibilities  of  both  simulator  designer  and  government  program 
manager  during  simulator  design. 

3.  Pre-  and  postsimulator  design  considerations  of  the  government  program 
manager. 

To  provide  information  in  these  areas,  discussions  are  divided  between: 


1.  Tasks  assigned  the  contractor. 

2.  Actions  required  of  the  government  program  manager. 

Discussions  of  the  contractor  and  government  program  manager  tasks  include 
specified  inputs  and  outputs.  Figure  1,  from  Development  Guidelines  (N),  illustrates  the 
steps  in  the  development  of  a  maintenance-training  simulator  specification.  These  steps 
are  described  in  the  following  paragraphs. 

Section  2.  Task  Identification 


This  section  discusses  the  procedure  of  identifying  maintenance  activities  to  be 
incorporated  in  the  trainer. 

Identification  of  maintenance  activities  requires  a  Preliminary  Task  Identification 
Matrix  (PTIM).  The  PT1M  is  developed  by  the  contractor  through  "brain  storming"  and  by 
observing  actual  maintenance  training  activities.  The  government  and  the  contractor 
cooperate  in  evaluating  and  validating  the  PTIM  to  produce  a  Task  Identification  Matrix 
(TIM).  During  validation,  changes  can  be  made  as  errors  or  omissions  are  identified  in  the 
task  list.  The  contractor's  task  analyst,  aided  by  system  hardware  experts,  also  prepares  a 
decision  criteria  list  during  validation.  Preparation  of  this  list  is  aided  by  an  analysis  of 
individual  tasks  to  define  special  behaviors  associated  with  specific  equipments.  Criteria 
for  this  analysis  of  individual  tasks  are  the  human  engineering  considerations  affecting  the 
design  of  controls,  labels,  work  space,  personnel  safety,  equipment  damage,  and  unique 
tools  and  test  equipment. 

Section  3.  Task  Analysis 

The  purpose  of  the  Task  Analysis  process,  as  noted  by  Development  Guidelines  (N),  is 
to  determine  the  task  practice  requirements.  Task  Analysis  begins  with  the  development 
of  a  Gross  Task  Analysis,  which  is  performed  by  entering  data  from  the  TIM  on  Gross  Task 
Data  sheets.  The  sheets  produce  a  document  that  is  complete  and  that  reflects  the 
maintenance  mission  in  terms  of  the  technician's  general  responsibility  for  each  task 
listed  in  the  TIM. 

The  contr=<~tor  examines  the  Gross  Task  Analysis  for  Normal  Repertoire  (NR)  and 
Special  Behavior  (SB)  required  to  perform  the  listed  maintenance  tasks.  The  equipment- 
related  special  behaviors  are  used  as  an  input  to  the  NR/SB  decision-criteria  development. 
The  NR  at  the  Gross  Task  Analysis  stage  of  the  Task  Analysis  might  be,  for  example,  the 
aptitude  of  a  maintenance  technician  to  perform  simple  manipulative  tasks  that  require 
only  finger  dexterity  and/or  hand-eye  coordination.  SB  might  be  the  skill  or  knowledge 
requiring  review  and  reinforcement--that  is,  additional  training  or  practice  to  achieve  the 
specified  level  of  task  performance.  Tasks  identified  as  NRs  are  dropped  from  the  task 
list  (and  any  further  analysis).  The  decision  to  drop  any  task  at  this  stage  of  the  task 
analysis  depends  on  whether  or  not  task  performance  involves  procedures  meeting  the  NR 
criteria.  Government  review  provides  final  approval  of  the  dropped  NRs. 

The  next  stage  of  the  Task  Analysis  is  a  detailed  step-level  analysis  performed  by  the 
contractor.  The  principal  difference  between  Gross  Task  Analysis  and  this  step-level 
analysis  is  the  number  of  steps  into  which  the  tasks  are  broken. 

The  last  stage  of  the  Task  Analysis  is  a  final  NR/SB  decision-and-division  procedure. 
The  criteria  for  producing  the  NR/SB  decision-and-division  procedure  are  based  on 
prerequisite  courses  taught  at  NAMTRADET,  Fleet  Readiness  Aircraft  Maintenance  Pro¬ 
grams  (FRAMPS),  "A,"  "B,"  and  "C"  schools.  The  contractor’s  first  action  at  this  stage  of 
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Task  Analysis  is  to  examine  the  FRAMP  prerequisite  training  curriculum  to  determine 
whether  any  of  the  practice  requirements  associated  with  the  task  occur  in  the 
prerequisite  training.  Those  that  occur  are  dropped  as  NRs.  Next,  the  contractor 
determines  the  training  activity  and  the  level  of  maintenance  for  which  the  simulator  is 
being  considered.  Tasks  not  applicable  to  the  activity  or  level  of  maintenance  are 
dropped.  Finally,  the  contractor  establishes  priorities  for  the  remaining  tasks,  using  as 
criteria  (1)  difficulty,  (2)  frequency,  (3)  importance,  and  (4)  level  of  proficiency  required. 
The  contractor's  actions  are  reviewed  by  the  government. 

Section  4.  Functional  Simulator  Design 

Of  all  the  steps  in  the  development  of  a  specification  for  a  maintenance-training 
simulator,  Functional  Simulator  Design  is  the  most  difficult.  Development  Guidelines  (N) 
states  that  ".  .  .  incorrect  or  incomplete  performance  of  this  step  will  produce  a  defective 
simulator.  We  have  attempted  to  proceduralize  portions  of  a  highly  creative  process. 
However,  the  task  .  .  .  cannot  be  reduced  to  a  square-filling  exercise.  For  these  reasons, 
we  recommend  that  this  step  be  performed  by  a  qualified  and  experienced  Instructional 
Development  technologist." 

In  Functional  Simulator  Design,  the  contractor  first  compares  the  skill  prerequisites 
for  the  job  against  the  skill  requirement  of  each  task  step  in  the  Task  Analysis.  If  the 
skill  is  new  or  a  new  application  of  an  old  one,  it  is  evaluated  in  terms  of  learning 
difficulty,  error  likelihood,  possible  equipment  damage,  or  possible  injury  to  the  task 
performer.  Any  of  these  criteria  make  the  skill  a  candidate  for  supervised  practice  in 
training,  and  simulation  is  considered.  The  candidates  for  supervised  practice  are 
reviewed  by  the  contractor  to  recommend  the  media  that  will  best  support  training. 
Development  Guidelines  (N)  states  that  "...  in  all  cases,  the  media  decisions  shall  be 
within  bounds  of  proven  simulation  techniques,"  and  continues  by  directing  that  the  media 
choices  shall  always  favor  the  simplest  technique  to  provide  the  necessary  practice. 
Government  review  is  used  to  validate  the  task  practice  and  methods/media  decisions. 

In  the  next  stage  in  the  Functional  Simulator  Design  step,  the  contractor  determines 
simulator  concepts  by  looking  for  commonality  in  tasks  or  task  steps  and  in  the  methods 
and  media  decisions.  Tasks  with  common  purposes  may  use  the  same  media  and 
instruction  methods.  The  idea  here  is  to  simulate  the  task  environment  as  accurately  as 
necessary  to  permit  realistic  task  practice  by  examining  the  practice  requirements  for 
commonality.  From  this  examination,  the  contractor  determines  the  number  of  different 
task-performance  (and  therefore  practice)  environments  required.  Development  Guide¬ 
lines  (N)  states  that  ".  .  .  if  there  are  several  practice  environments,  thought  shall  be 
given  to  individual  or  part-task  simulators.  Multiple  tasks  performed  in  the  same  physical 
environment  are  logical  choices  for  inclusion  in  a  multi-purpose  simulator;  however,  class 
size,  practice  length,  and  simulator  availability,  or  the  number  of  simulators  required, 
shall  be  considered.  Generally  speaking,  individual  task  (part-task)  trainers  are  more 
flexible  from  an  availability  standpoint  and  are  more  cost  effective  to  produce."  Another 
consideration  to  be  weighed  when  deciding  simulator  type  is  the  suitability  of  a  given 
simulator  for  the  kind  of  learning  involved. 

The  final  action  in  determining  simulator  concepts  is  that  once  the  number  of 
different  simulators  and  the  general  purposes  of  each  are  established,  the  contractor 
establishes  what  (functionally)  is  to  be  simulated.  For  each  simulator,  the  physical 
characteristics  of  the  actual  hardware  and  the  environment  with  which  the  task  performer 
interacts  during  task  performance  are  listed.  Development  Guidelines  (N)  states: 

The  contractor  shall  then  prepare  a  script  for  task  performance 

(practice).  The  script  shall  be  written  at  the  individual  behavioral 
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action  level.  Each  script  step  shall  contain  three  separate  format 
elements:  student,  simulator,  and  instructor.  The  student  activity  shall 
be  described.  The  simulator  condition  or  response  shall  be  described 
completely  by  noting  the  condition  or  state  of  each  previously  listed 
physical  characteristic.  The  instructor's  role  in  task  practice  shall  be 
specified  in  terms  of  monitoring,  feedback,  or  other  instructor  func¬ 
tions. 

The  final  stage  in  the  Functional  Simulator  Design  step  is  the  drafting  and  review  of 
the  actual  specification.  The  specification  includes: 

1.  The  Functional  Requirements  that  entail,  in  addition  to  the  scripted  task 
practice  sequence,  a  general  description  of  the  simulator,  and  an  overview  of  the 
simulator,  its  operation,  and  its  use. 

2.  The  sensory  (aural,  visual,  tactile,  etc.)  fields  of  the  simulator,  on  a  component- 
by-component  basis. 

After  the  specification  has  been  completed,  the  government  reviews  it  to  ensure  that 
it  is  correct. 

Section  5.  Post-Specification  Activities 

This  short  section  discusses  the  desired  use  of  specification  contractor  as  a  technical 
advisor  to  the  program  manager  during  the  acquisition  of  the  maintenance  training 
simulator  hardware. 

Section  6.  Pre-Simulator  Functional  Specification  Contract  Award 

This  section  presents  basic  information  requirements,  general  organizational 
principles  to  follow  within  the  government's  maintenance  training  simulator-development 
program,  and  suggested  inclusions  for  a  functional  specification  development  contractor's 
statement  of  work.  The  suggested  statement  of  work  is  composed  of  the  previous  text 
and  diagrams  contained  in  Development  Guidelines  (N). 

Phase  1  Document  (N) 

The  Phase  I  Document  (N)  deals  with  a  specific  area  of  simulator  development;  that 
is,  the  development  of  a  NAMTRADET  organizational-level  electronics  maintenance¬ 
training  simulator. 

Section  1,  Introduction 

The  introduction  provides  a  brief  background  of  simulator  development  responsibili¬ 
ties  in  the  Naval  Air  System  Command  (NAVAIRSYSCOM)  subordinate  organizations. 

A  section  entitled  "Plan  to  Develop  Guidelines"  describes  the  steps  to  be  followed  by 
a  project  director  in  procuring  a  maintenance  training  simulator. 

The  Phase  I  Document  (N)  states  that  experience  with  maintenance-training 
simulators  currently  in  the  field  suggests  the  following  deficiencies  and  omissions: 

1.  Task  analyses  underlying  the  requirements  for  a  simulator  were  sometimes 
performed  inadequately.  Consequently,  tasks  that  should  have  been  taught  were  not  being 
taught,  incorrect  procedures  were  being  taught,  and  capabilities  of  simulators  were  not 
being  used. 


2.  Simulator  design  and  development  were  inadequately  integrated  with  the  course 
curriculum.  Often,  the  simulator  was  simply  an  added  aid  to  training. 

3.  Documentation  to  assist  the  instructor  and  the  student  in  using  the  simulator  was 
often  fragmentary.  As  a  consequence,  the  simulator  was  not  used  to  its  fullest,  and 
sometimes  not  at  ail. 

k.  Instructor  training  was  sometimes  overlooked. 

5.  Provision  for  the  prompt  maintenance  of  the  simulator  was  insufficient.  Con¬ 
sequently,  when  needed,  the  simulator  was  sometimes  unavailable.  Maintenance  manuals 
were  also  lacking. 

The  Phase  I  Document  (N)  responds  to  these  deficiencies  and  omissions  by  stating 
that  most  of  them  could  receive  appropriate  treatment  in  a  well-written  specification  for 
a  simulator  manufacturer.  However,  because  of  the  importance  of  task  analysis,  as 
indicated  by  the  deficiencies  and  omissions,  it  should  receive  thorough  treatment  separate 
from  the  simulator  specification.  Thus,  procurement  should  be  a  two-part  process--part 
one,  the  task  analysis;  part  two,  the  manufacturer's  specification.  Task  analysis  might  be 
performed  more  rigorously  and  with  less  bias  by  other  than  a  simulator  manufacturer  (a 
specifications  contractor  or  government  agency  specializing  in  training). 

Section  2.  Phase  I  Task  Analyses  and  Simulator  Configuration 

The  following  actions  of  the  maintenance-training-simulator  project  director  during 
the  task  analysis  and  simulator  configuration-development  phase  are  discussed: 

l.  The  initiation  of  the  project  by  the  issue  of  an  AIRTASK  from  the  funding 
agency  to  a  performing  agency. 

2.  The  need  for  delineation  of  the  constraints  on  the  project  and  the  authority  and 
responsibilities  of  the  performing  agency. 

3.  The  general  steps  to  be  performed  by  the  project  director  in  coordinating, 
scheduling,  and  monitoring  "Phase  I"  work  among  the  various  possible  military  and  civilian 
organizations. 

4.  The  collection  of  relevant  information  before  letting  a  "Phase  I  work"  contract. 
This  information  is  needed  by  the  bidding  contractors  to  estimate  precisely  the  time  and 
cost  required  to  perform  that  work.  Sample  forms  that  might  be  used  for  the  collection 
of  these  data  are  also  given. 

5.  The  possible  decisions  the  project  director  could  be  required  to  make  about  task 
analysis  data  and  what  the  actual  "Phase  I  work"  will  entail. 

6.  The  actual  preparation  of  a  "procurement  package"  for  the  "Phase  I  work." 

7.  An  introduction  to  the  sample  statement  of  work  (SOW)  included  in  the  third 
section  of  the  Phase  I  Document  (N). 

Section  3.  Statement  of  Work 


A  sample  statement  of  work  (SOW)  for  the  performance  of  "Phase  1  work"  by  a 
maintenance-training-simulator  specifications  contractor  is  presented.  Although  this 


sample  SOW  is  conceptually  the  same  as  that  in  Development  Guidelines  (N),  the  exact 
steps  and  their  relative  order  differ.  The  steps  are  presented  below,  with  indications  of 
where  they  differ  from  those  in  Development  Guidelines  (N). 

1.  Preliminary  Task  Identification  Matrix  (PTIM).  The  specifications  contractor, 
using  documents  provided  by  the  government  and  other  documents,  prepares  a  PTIM.  The 
development  and  format  of  the  PTIM  is  the  same  as  in  Development  Guidelines  (N). 

2.  Preliminary  Task  List  for  Organizational-Level  Maintenance  (PTLOLM).  The 
contractor  prepares  the  PTLOLM  from  the  PTIM  by  identifying  only  organizational-level 
maintenance  tasks.  Each  task  is  listed  with  the  hardware  item  on  which  the  task  is 
performed  and  is  described  briefly  in  terms  familiar  to  those  who  will  review  the 
PTLOLM.  This  action  is  not  specifically  addressed  in  Development  Guidelines  (N). 

3.  Validation  of  PTLOLM.  The  specifications  contractor  will  visit  a  maintenance 
performance  site  to  validate  the  maintenance  and  to  discuss  the  maintenance  tasks  with 
technicians.  Discrepancies  are  noted,  along  with  the  names  of  the  maintenance 
technicians.  This  validation  process  might  also  reveal  additional  organizational-level 
maintenance  tasks. 

4.  Government  review  of  the  PTLOLM.  The  government  reviews  the  contractor's 
efforts  at  this  stage  of  the  specification  development,  as  was  done  in  Development 
Guidelines  (N). 

5.  Gross  Task  Analysis  (GTA).  From  the  validated  and  reviewed  PTLOLM,  the 
contractor  performs  a  Gross  Task  Analysis,  as  was  done  in  Development  Guidelines  (N). 

6.  Relative  Importance  of  Tasks.  To  develop  the  GTA,  the  contractor,  using  Navy 
and  civilian  personnel  intimately  acquainted  with  the  maintenance  of  the  system  under 
study,  collects  and  interprets  data  on  the  relative  importance  of  the  tasks.  The  relative 
importance  of  tasks  is  determined  by  criteria  established  in  the  Interservice  Procedures 
for  Instructional  Systems  Development,  Phase  I,  Analysis  (NAVEDTRA  106A,  1  August 
1975).  Development  Guidelines  (N)  establishes  task  importance  in  the  "Detailed  Simulator 
Concepts"  determination  process. 

7.  Commonality  of  Tasks.  The  specifications  contractor  classifies  into  sets  those 
tasks  that  require  the  same,  or  very  similar,  behavior  with  the  same  class  of  equipment. 
From  each  set  of  common  tasks,  the  contractor  selects  sample  tasks  to  represent  the 
whole  set.  Commonality  of  Tasks  is  determined  during  the  "Detailed  Simulator  Concepts" 
process  in  Development  Guidelines  (N). 

8.  Selection  of  Tasks  for  Training  at  a  NAMTRADET.  Results  of  the  GTA, 
Relative  Importance  of  Tasks,  and  Commonality  of  Tasks  are  examined  by  the  contractor 
and  the  tasks  are  classified  as  (1)  tasks  to  be  taught  at  a  NAMTRADET,  (2)  tasks  to  be 
taught  at  a  FRAMP  school,  and  (3)  tasks  that  need  not  be  taught  at  all  because  the 
student  will  have  already  acquired  the  knowledge  and  skills  at  "A"  school.  Action  of  this 
type  is  not  provided  by  Development  Guidelines  (N). 

9.  Government  Review.  Another  government  review  at  this  stage  covers  the  GTA, 
Relative  Importance  of  Tasks,  Commonality  of  Tasks,  and  Selection  of  Tasks  for  Training 
at  a  NAMTRADET,  A  similar  review  is  required  by  Development  Guidelines  (N). 

10.  Final  Task  List  (FTL)  for  a  NAMTRADET.  The  review  previously  executed 
results  in  a  Final  Task  List  (FTL)  tasks  for  NAMTRADET  training.  Development 
Guidelines  (N)  does  the  same.  Both  FTLs  are  used  in  selecting  media. 
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11.  Detailed  Task  Analysis  (DTA).  From  the  FTL,  the  contractor  performs  a 
Detailed  Task  Analysis  (DTA)  describing  in  minute  detail,  the  steps  in  each  task.  This 
step  is  the  same  as  in  Development  Guidelines  (N). 

12.  Behavioral  Objectives  (BO).  Based  upon  the  DTA,  the  contractor  will  develop 
behavioral  objectives.  Each  of  these  behavioral  objectives  includes  (a)  an  identification 
and  description  of  the  behavior,  (b)  a  description  of  the  relevant  conditions  under  which 
the  behavior  is  to  occur,  and  (c)  a  definition  of  criteria  of  acceptable  performance. 
Development  Guidelines  (N)  does  not  provide  for  the  development  of  Behavioral  Objec¬ 
tives,  although  it  does  provide  for  the  development  of  NR/SB  criteria  to  drop  tasks  that 
are  not  required  in  the  DTA. 

13.  Tasks,  Types  of  Learning,  Method/Media  Selection.  Using  the  DTAs  and  BOs 
for  each  task  or  task  step,  the  contractor  classifies  the  tasks  into  categories  representing 
"types  of  learning,"  following  some  specific  scheme  in  the  literature  of  educational 
psychology.  The  contractor  provides  a  reference  for  the  classification  scheme,  and 
defines  and  illustrates  each  type  of  learning.  The  contractor  recommends  methods/media 
suitable  for  training  each  task,  and  also  classifies  the  tasks  into  thos<  to  be  trained  on  a 
simulator  and  those  to  be  trained  by  other  means.  Development  Guidelines  (N)  addresses 
selection  of  media  but  does  not  address  selection  of  materials  to  be  taught  by  other 
means. 

14.  Preliminary  Simulator  Designs.  Based  on  the  previously  performed  step,  the 
contractor  proposes  alternative  designs  for  accomplishing  the  desired  training.  Each 
design  will  be  illustrated,  and,  for  each  design,  the  contractor  briefly  describes  the 
advantages  and  disadvantages,  the  fidelity,  and  the  expectations  about  the  transfer  of 
training.  Also,  the  contractor  provides  rough  estimates  of  tne  cost  of  alternative  designs, 
and  an  estimate  of  the  cost  for  5  years.  Development  Guidelines  (N)  provides  a  step  for 
the  determination  of  simulator  design,  but  requires  the  contractor  to  identify  only 
alternative  simulator  designs,  omitting  the  determination  of  advantages/disadvantages, 
the  fidelity,  expectations  about  the  training  transfer,  and  relative  costs. 

15.  Government  Review.  A  government  review  is  again  provided  at  this  stage  of 
specification  development.  This  review  will  result  in  the  selection  of  a  maintenance 
training  simulator  design.  Development  Guidelines  (N)  provides  for  a  similar  review. 

16.  Final  Simulator  Design.  The  contractor  will  specify  the  final  simulator  design, 
the  physical  and  preceptual  appearance,  and  the  function  of  the  components.  In  these 
descriptions,  the  contractor  is  required  to  specify  what  is  to  be  represented,  not  how  the 
representation  will  be  accomplished  by  a  maintenance-training-simulator  manufacturer. 

The  contractor  is  required  to  include,  in  the  final  simulator  design,  a  descrip¬ 
tion  of  how  the  student  will  interact  with  the  maintenance-training  simulator  and  how  the 
student  will  be  provided  performance  feedback.  The  instructor's  function  will  be 
incorporated  in  the  design.  If  student  and  instructor  panels  are  required  by  the  design,  the 
specifications  contractor  is  required  to  provide  a  functional  design  for  such  panels. 

Final  Simulator  Design  is  detailed  in  Development  Guidelines  (N)  under  "De¬ 
tailed  Simulator  Concepts." 

17.  Script  for  Task  Performance.  The  specifications  contractor  prepares  a  script 
describing  task  performance  in  achieving  the  behavioral  objectives.  The  script  is  required 
to  address  the  roles  of  the  student,  the  instructor,  and  the  maintenance-training 
simulator.  The  contractor  is  also  required  to  modify  the  behavioral  objectives,  as 


necessary,  and  integrate  the  modified  behavioral  objectives  with  the  script  and  with  the 
simulator  design.  Development  Guidelines  (N)  calls  for  script  development  under 
"Detailed  Simulator  Concepts." 

18.  Final  Government  Review.  A  final  government  review  of  the  specifications 
contractor  work  concludes  the  sample  SOW.  The  results  of  this  review  are  used  in  the 
production  of  an  actual  procurement  specification  for  a  maintenance-training  simulator. 
The  final  review  provided  in  the  process  detailed  in  Development  Guidelines  (N)  occurs 
after  a  sample  specification  has  been  developed  by  the  specifications  contractor. 

The  sample  SOW  in  the  Phase  I  Document  (N)  describes  the  contractor's  steps  in 
paragraphs  that  detail  the  performance  requirements.  Data  entries  making  the  sample 
SOW  applicable  to  a  specific  system  under  study  are  set  in  parentheses.  Remarks  to 
orient  the  administrator  (project  director)  or  to  justify  a  suggested  course  of  action  are 
contained  in  brackets  and  are  preceded  by  the  word  ADMINISTRATOR.  The  sample  SOW 
concludes  with  sample  Contract  Data  Requirements  Lists  (CDRLs,  DD  Form  1423)  and 
Data  Item  Descriptions  (DIDs,  DD  Form  1664),  which  relate  to  specific  specification- 
contractor  performance  areas. 

Section  4.  Miscellaneous  Advice  to  the  Project  Director 

Brief  comments  to  the  project  director  are  included  in  this  section,  covering  such 
areas  as  funding,  schedules,  reviews,  and  simulator  manufacturer  queries.  Of  particular 
note  are  guidelines  and  recommendations  to  assist  in  determining  specification-suggested 
features  or  capabilities  that  are  not  really  needed. 

Flow  Diagram 

The  Phase  I  Document  (N)  concludes  with  a  flow  diagram  that  relates  the  order  and 
interrelation  of  steps  in  developing  a  maintenance-training  simulator  specification.  The 
flow  diagram,  which  is  provided  in  Appendix  B,  divides  the  actions  of  the  project  director 
and  the  specifications  contractor. 

Phase  II  Document  (N) 


The  Phase  II  Document  (N)  addresses  the  procurement  of  a  maintenance-training 
simulator  system  for  organizational-level  electronics  maintenance  training  at  NAMTRA- 
DETs. 

Section  1.  Introduction 


The  Introduction  of  the  Phase  II  Document  (N)  states  that  this  document,  with  Phase  I 
Document  (N),  provides  a  two-phase  approach  to  the  development  and  procurement  of  an 
organizational-level  electronics  maintenance-training  simulator  for  a  NAMTRADET.  The 
two-phase  approach  is  based  on  the  assumption  that  the  most  critical  aspect  of  developing 
a  maintenance-training  simulator  system  is  the  task  analysis,  and  the  task  analysis  would 
probably  be  best  performed  by  someone  other  than  a  maintenance-training  simulator 
manufacturer.  The  task  analysis  of  Phase  I  Document  (N)  serves  as  the  main  input  to  a 
procurement  specification  described  in  the  Phase  II  Document  (N).  The  following  data 
items  from  the  Phase  I  Document  (N)  procedures  serve  as  the  input  to  the  Phase  II 
Document  (N). 

1.  A  list  of  tasks  to  be  trained  at  a  NAMTRADET. 

2.  A  detailed  task  analysis  for  each  task. 


3.  The  methods/media  suitable  for  training  each  task. 

4.  Behavioral  objectives  for  the  tasks. 

5.  The  simulator  configuration. 

6.  A  script  for  task  performance  adressing  the  roles  of  the  student,  the  instructor, 
and  the  maintenance  training  simulator. 

The  Phase  II  Document  (N)  states  that  its  functions  are  to  provide  general  discussions 
and  advice  to  the  project  director  in  the  actual  procurement  of  a  maintenance-training 
simulator,  such  as  planning  and  contractor  contact,  and  to  provide  a  sample  procurement 
specification. 

Section  2.  Procurement  Planning 

A  brief  general  discussion  covers  both  the  formal  and  informal  planning.  A  format 
for  an  Illustrative  Procurement  Plan  is  provided  to  assist  the  government  program 
manager.  A  table  is  also  included  detailing  time  required  for  procurements  at  NADC. 

Section  3.  Contact  with  Contractors 


The  allowable  contact  of  government  employees  with  contractors  is  described. 
Comments  on  debriefing  of  unsuccessful  offerors  and  technical  direction  of  award 
contractor  are  also  included. 

Section  4.  General  Information  on  Specifications 

General  definitions  of  specification  types  and  other  procurement  specification  topics 
are  provided  for  the  project  director.  Requirements  to  be  included  in  procurement 
specifications  are  briefly  covered,  and  a  checklist  of  specifications  requirements  from 
MIL  STD  961  is  provided. 

Section  5.  Sample  Specification 

The  bulk  of  the  Phase  II  Document  (N)  is  devoted  to  the  presentation  of  this  sample 
specification  for  the  procurement  of  a  maintenance-training  simulator.  The  Phase  II 
Document  (N)  reports  that  an  attempt  is  made  in  the  sample  specification  to  include  those 
areas  typically  covered  in  a  simulator  specification- -the  engineering  and  integrated 
logistics  support  "boilerplate." 

In  the  document,  the  specification  is  presented  on  the  left-hand  pages;  and  informa¬ 
tion,  explanations,  and  instructions  for  the  specification  writer,  on  the  right-hand  pages. 
The  specification  also  provides  instructions  for  inclusion  of  data  generated  by  Phase  I 
Document  (N). 

Section  6.  Parts  and  Sections  of  the  Request  for  Proposals  (RFP) 

The  sample  specification  is  only  part  of  the  actual  RFP  needed  for  procurement  of 
the  maintenance-training  simulator.  This  section  of  the  Phase  II  Document  (N)  provides 
the  schedule  (in  the  form  of  Contract  Data  Requirements  Lists,  CDRLs,  DD  Form  142^) 
that  accompanies  the  specification  (actually  part  of  section  6  of  a  specification).  The 
schedule  includes  the  deliverable  items,  such  as  Data  Item  Descriptions  (DIDs,  DD  Form 
1664),  as  well  as  information  on  other  RFP  requirements. 


Section  7.  Technical  Proposal  Requirements,  Proposal  Evaluation  Plan,  and  Proposal 
Evaluation  Reports 

Material  on  the  Technical  Proposal  Requirements  (TPR),  the  Proposal  Evaluation  Plan 
(PEP),  and  on  the  Proposal  Evaluation  Report  (PER)  is  presented. 

Computer  System  for  Simulator  System 


The  sample  specification  (section  5)  did  not  include  specifications  on  a  computer 
system  for  the  simulator.  This  enclosure  to  the  Phase  II  Document  (N)  presents 
statements  on  computer  systems  that  might  be  used  in  a  specification.  An  outline  of 
functional  requirements  for  microcomputer  applications  and  a  document,  written  in  the 
form  of  a  specification,  covering  computer  software  for  a  fairly  complex  computer 
system  are  included.  The  computer  software  specification  is  written  with  the  assumption 
that  FORTRAN  will  be  used  as  the  major  programming  language. 


THE  AIR  FORCE  STUDY 

The  Air  Force  Study  at  present  consists  of  the  following  two  documents,  prepared  for 
the  Air  Force  Human  Resources  Laboratory  (AFHRL),  Lowry  Air  Force  Base,  CO  by 
Applied  Science  Associates  (ASA),  Inc.,  under  contract  F33615-78-C-0019. 

1.  ISP  and  SPO  Procedural  Requirements  for  the  Design  and  Acquisition  of 
Maintenance-Training  Simulators. 

2.  Maintenance  Training  Simulator  Design  and  Acquisition  Model  Specifications. 

Hereafter,  these  documents  will  be  referred  to  as  Procedural  Requirements  (AF)  and 
Model  Specifications  (AF)  respectively. 

The  two  Air  Force  reports  define  different  program  objectives.  Procedural  Require¬ 
ments  (AF)  covers  the  collection,  analysis,  and  documentation  of  information  for  the 
design,  fabrication,  and  life-cycle  maintenance  of  maintenance  simulators.  The  Model 
Specifications  (AF)  covers  the  acquisition  of  baseline  knowledge  on  techniques,  proce¬ 
dures,  and  principles  necessary  for  broad  applications  of  simulations  in  maintenance 
training.  The  latter  objective  appears  more  in  line  with  the  needs  of  this  study. 

The  documents  are  preliminary  drafts  and  the  final  drafts  are  still  in  preparation. 
The  following  paragraphs  provide  a  synopsis  of  their  contents  using  the  headings  of  the 
documents  and  their  organization  (see  Appendix  C).  Statements  and  opinions  are  those  of 
the  authors  and  are  presented  without  comment. 

Procedural  Requirements  (AF) 

Section  1.  Introduction 

Procedural  Requirements  (AF)  reports  that  the  initial  phase  of  the  Air  Force  effort 
focused  on  the  information  and  procedures  required  by  Instructional  System  Development 
(ISD)  teams  and  Systems  Program  Office  (SPO)  Training  Equipment  Acquisition  Managers 
in  developing  cost-effective  technical  training.  Subsequent  phases  define  the 
maintenance-training  simulator  from  maintenance  task  data,  model  training  equipment, 
and  model  procurement  specifications. 
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The  current  procedures  for  ISD  training-equipment  design  and  SIM  training-equip¬ 
ment  acquisition  are  reviewed.  The  review,  which  is  a  summary  of  many  surveys,  includes 
an  interpretation  of  the  project  findings  in  terms  of  technological  gaps  and  problem  areas 
faced  in  designing  a  cost-effective  maintenance-training  simulator.  A  training-equipment 
design-process  model  is  included  to  serve  as  the  framework  for  design  decisions. 

Section  2.  Current  Training  Equipment  Procurement  Process 

An  overview  of  the  complete  process  of  acquiring  maintenance-training  equipment  by 
the  Air  Force  is  given. 

Present  ISP  Training  Equipment  Process.  Individual  Air  Force  Command  and 
interservice  procedures  for  the  ISD  process  have  been  developed.  All  these  ISO  processes 
have  a  common  flow,  entailing  the  following  actions: 

1.  Activities  requiring  training  analyzed. 

2.  Training  requirements  determined. 

3.  Training  objectives  and  their  sequence  established. 

4.  Performance  measurement  techniques  developed. 

5.  Methods  and  media  selected. 

6.  Instructional  materials  developed. 

7.  The  instructional  program  conducted  and  evaluated. 

The  document  states  that  the  3306th  Test  and  Evaluation  Squadron  (T&ES)  has  been 
successful  in  applying  the  ISD  guidelines  to  the  design  of  maintenance-training  equipment. 
Therefore,  the  Air  Force,  in  developing  standardized  ISD  procedures,  has  adopted  this 
3306th  T&ES  ISD  process  for  evaluation. 

The  ISD  processes,  as  indicated  in  the  AF  documents,  are  not  significantly  different 
when  designing  training  equipment  for  existing  systems  or  for  new  systems,  or  for 
intermediate  or  organizational-level  tasks.  To  perform  effective  ISD,  comprehensive  task 
information  must  exist  or  be  generated. 

Manning  and  Training  of  ISD  Teams.  Personnel  selected  as  future  instructors  are 
assigned  as  ISD  team  members  in  the  3306th  T6cES.  The  Air  Force  philosophy  is  that  "it  is 
more  effective  to  select  and  train  individuals  to  be  ISD  analysts  who  are  already 
experienced  and  capable  instructors  and  also  experts  in  the  subject  matter  (Subject 
Matter  Experts)  (SMEs)  in  specific  types  of  subsystems  and  specialities,  than  it  is  to  select 
experienced  analysts  and  attempt  to  make  systems  specialists  and  competent  instructors 
of  them." 

The  various  steps  of  the  ISD  process  are  grouped  into  five  general  tasks  to  form  a 
model  for  ISD.  These  tasks  are  seldom  performed  in  sequential  order.  Figure  2  shows  how 
the  3306th  T&ES  procedural  steps  relate  to  the  general  ISD  model. 

The  ISD  descriptions  currently  in  use  by  the  Air  Force  were  reviewed,  and  three 
major  shortcomings  of  the  present  ISD  procedures  were  identified.  These  shortcomings, 
to  be  corrected  by  the  development  of  the  Training  Equipment  Design  Process  Model,  are: 

1.  The  present  procedures  describe  an  idealistic  data  availability  situation.  The 
depth,  accuracy,  and  reliability  of  the  task  data  that  is  usually  available  to  the  ISD 
analyst  does  not  permit  clean  abstraction  into  specific  categories  of  the  analysis 
procedure. 
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Figure  2.  Relations  between  general  model  and  3306th 
T&ES  procedural  steps. 


2.  The  ISD  guides  and  handbooks  available  are  "principle"  oriented  and  provide 
minimum  guidance  on  the  procedural  or  mechanistic  application  of  these  principles  in 
making  training  and  training  equipment  design  situations.  There  are  no  real  step-by-step 
"how  to"  guides. 

3.  1ST)  required  decision-making  necessitates  the  learning,  retention,  and  integra¬ 
tion  of  a  large  number  of  complex  concepts/constructs  and  associated  knowledges.  The 
ISD  procedures,  in  total,  constitute  a  set  of  skills  that  require  extensive  practice  for 
mastery. 

The  SPO  Training  Equipment  Acquisition  Process.  The  current  maintenance  training 
equipment  procurement  processes  in  the  Air  Force  do  not  fall  into  a  "typical  pattern,"  and 
are  not  consistent  throughout  the  Air  Force.  The  purpose  in  examining  the  SPO 
procedures  was  to  ".  .  .  study,  survey,  analyze,  and  determine  the  requirements  of  SPO 
Training  Equipment  Acquisition  Managers  who  are  charged  with  the  analysis  and  review  of 
simulation  requirements,  and  the  synthesis  of  those  requirements  into  procurement 
specifications  for  maintenance  simulators."  Data  sources  included  document  review,  Air 
Force  personnel  interviews,  and  Air  Force  directives  review. 

SPO  Training  Equipment  Acquisition  Manager  activities  fall  into  the  following 
categories: 

1.  Validation  of  training-device  functional  requirements  resulting  from  the  ISD 
process. 

2.  Validation  of  weapon-system  contractors'  engineering  data. 

3.  Preparation  of  procurement  specifications  that  translate  previously  validated 
results  into  hardware  design. 

4.  Management  of  contract  procedures  from  contractor  selection  through  training- 
equipment  development  to  ensure  timely  delivery  of  an  adequate  product. 

The  Air  Force  requires  the  SPO  acquisition  manager  to  participate  in  all  the 
analytical  and  review  processes  leading  to  the  ISD  team  specification  of  functional 
requirements.  When  final  requirements  are  presented  for  procurement,  the  SPO  acquisi¬ 
tion  manager  is  knowledgeable  about  the  system,  and,  specifically,  the  ISD  process  that 
has  generated  the  simulator  requirements.  The  SPO  acquisition  manager  can  contribute 
to  both  the  validation  of  device  functional  requirements  and  the  translation  of  those 
requirements  into  procurement  specifications. 

Major  Gaps  and  Problem  Areas.  Three  major  classes  of  problem  areas  and  technical 
gaps  to  be  addressed  and  corrected  by  the  SPO  Training  Equipment  Acquisition  Model  are 
defined.  This  will  serve  as  the  basis  for  developing  standardized  SPO  Training  Equipment 
Acquisition  procedures.  Problem  areas  and  technical  gaps  are: 

1.  Inappropriate  specifications. 

2.  Late  acquisition  (training  equipment  delivery  too  late  to  support  initial  operation 
training). 

3.  Variable  management  practices  (no  single  point  of  procurement  management). 


Section  3.  Training  Equipment  Design  Process  Model 

This  section  (1)  examines  the  training  equipment  (maintenance-training  simulator) 
design  process,  (2)  identifies  the  underlying  decision  logic  that  should  structure  the 
process  of  getting  from  job  information  to  simulator  design  characteristics,  and  (3) 
identifies  at  least  general  classes  of  information  that  are  really  required  to  support  each 
decision  set.  The  approach  used  was  to  adopt  a  general  model  with  decision  sets  described 
at  a  level  that  generalizes  across  most  situations  in  which  the  ISD  process  would  result  in 
a  maintenance-training  simulator. 

The  purpose  of  this  design  process  is  to  lay  the  groundwork  for  proposing  hierarchical 
and  associative  relationships  between  information  about  tasks  and  appropriate  training- 
equipment  functional  characteristics.  The  model  goes  beyond  existing  Air  Force  ISD 
procedures  and  is  intended  to  lead  toward: 

1.  Determination  of  appropriate  ISD  procedure  modifications  that  will  support  cost- 
effective  simulator  development. 

2.  Identification  of  the  procedural  steps  that  can  profit  by  a  reference  manual 
(handbook). 

3.  Description  of  documents  that  define  ISD  maintenance-simulator  requirements. 

The  following  decision  areas  span  the  evolution  of  training  equipment  functional 
characteristics  from  determining  what  must  be  accomplished  on  the  job  to  prescribing  the 
best  method  of  documenting  the  design: 

1.  Required  job-relevant  skills  and  knowledge. 

2.  Skills  and  knowledge  that  must  be  learned  by  the  trainees. 

3.  Skills  and  knowledge  that  can,  at  least  in  part,  be  learned  most  effectively 
through  practice  on  an  item  of  training  equipment. 

4.  Group  skills  and  knowledge  by  class  or  type  of  training  equipment. 

5.  For  each  type  of  training  equipment,  the  extent  to  which  skills  and  knowledge 
must  be  learned. 

6.  The  order  in  which  specific  classes  of  training  equipment  should  be  employed  to 
facilitate  learning. 

7.  The  design  concepts  that  constitute  preliminary  descriptions  of  training  equip¬ 
ment  characteristics  to  most  effectively  support  specific  learning  requirements. 

8.  A  preliminary  Plan  of  Instruction  (POI)  that  integrates  training  equipment  and 
all  other  appropriate  media  into  an  effective  training  scenario. 

9.  Equipment  design  and  detail  revised  and  completed  for  all  relevant  functional 
characteristics  to  be  used  in  the  training  scenario. 

10.  Required  training  equipment  functional  characteristics  documented  as  the 
principal  inputs  to  the  SPO  acquisition  process. 
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Appendix  D  shows  this  decision  process  and  summarizes  the  principal  informational 
inputs  necessary  for  its  execution. 

Section  4.  Implications  for  Instructional  Systems  Development 

This  section  pinpoints  those  aspects  of  training-equipment  selection  and  design  to  be 
dealt  with  in  an  ISD  handbook  and  model  specifications.  These  aspects  were  derived  by 
examining  the  current  3306th  T&ES  ISD  process  for  selection  and  design  of  maintenance 
trainers,  guided  by  the  Training  Equipment  Design  Process  Model. 

In  cases  where  ISD  is  applied  in  determining  training  and  training  equipment  by  the 
Air  Force,  the  results  are  better  than  other  methods  previously  applied.  Four  areas  that 
provide  the  opportunity  for  improvement  of  the  present  ISD  procedures  used  by  the  3306th 
T&ES  are: 

1.  Procedures  for  training-equipment  selection. 

2.  Procedures  for  maintenance-simulator  design. 

3.  Specification  for  maintenance-simulator  design  documentation. 

4.  ISD  team  training. 

Procedures  for  Training  Equipment  Selection.  A  maintenance-trainer  simulator 
should  be  selected  as  the  preferred  training  equipment  only  when  it  provides  the  most 
cost-effective  training.  The  Procedural  Requirements  (AF)  deals  with  the  selection  of  all 
types  of  training  equipment  and  course  materials,  not  just  maintenance-training 
simulators.  Systematic  procedures  and  formalized  guidance  for  selecting  the  most  cost- 
effective  training  system  will  be  developed  and  included  in  the  ISD  handbook.  The 
procedures  are  intended  to  replace  current  use  of  Rationale  Checklists  and  Media 
Analograms.  Specific  procedures  will  be  heavily  based  on  taxonomic  relations  being 
identified  in  the  Air  Force  study.  These  procedures  will  meet  the  following  general 
criteria: 


1.  They  must  be  as  mechanistic  as  possible.  Therefore,  SMEs  (ISD  team  analysts) 
without  extensive  experience  and/or  formalized  training  in  human  factors  and  educational 
psychology  will  be  able  to  apply  them  uniformly  and  reliably. 

2.  They  will  be  usable  within  the  ISD  process  currently  being  used  by  the  Air  Force. 

3.  Documentation  (handbooks  and  procedures)  must  be  acceptable  to  SMEs. 

4.  The  handbook  must  be  applicable  to  both  organizational-  and  intermediate-level 
tasks. 

Procedures  for  Maintenance  Simulator  Design.  Maintenance-training  simulator 
design  decisions  at  the  functional  level  are  numerous  and  complex.  Formalized  simulator 
design  procedures  have  been  lacking,  and  there  is  a  tendency  by  Air  Force  commands  to 
have  simulator  characteristics  that  closely  duplicate  the  actual  equipment.  Very  little 
attention  is  paid  in  the  Air  Force  at  present  to  learning  enhancements  through  specialized 
instructional  features.  By  building  human-factors  expertise  into  job-performance  aids 
available  to  the  ISD  team  analysis,  this  difficulty  can  be  corrected. 

Specifications  for  Maintenance  Simulator  Design  Documentation.  A  section  of  the 
ISD  handbook,  to  be  prepared,  will  focus  on  procedures  for  developing  functional 
descriptions  of  maintenance-training  simulators.  This  portion  of  the  ISD  handbook  will: 


1.  Present  a  comprehensive  model  specification  as  an  output  of  the  ISD  process  for 
documenting  maintenance-training  simulators. 

2.  Define  all  data  items  so  that  the  criteria  for  necessary  information  are  clear  and 
comprehensive  enough  to  accommodate  all  conceivable  varieties  of  maintenance-training 
simulators  for  Air  Force  use. 


3.  Provide  instructions  for  selecting  the  appropriate  data  categories  to  meet 
description  requirements  for  any  particular  maintenance-training  simulator. 

4.  Guide  the  formatting  and  detailed  information  content  of  the  complete 
maintenance-training  simulator  functional  description. 

The  completed  handbook  will  include  (1)  step-by-step  procedures,  at  a  level  of  detail 
compatible  with  inexperienced  ISD  analysts,  and  (2)  reference  tables  and  matrices  that 
relate  all  significant  dimensions  of  the  decision  process.  It  will  be  configured  so  that  it 
will  be  useful  to  ISD  team  analysts  unfamiliar  with  total  ISD  procedures  as  both  a 
procedural  guide  and  reference  source.  These  configurations  will  also  serve  as  an  easy 
and  quick  reference  source  on  task-equipment  relations  for  experienced  analysts. 

ISD  Team  Training.  To  facilitate  use  of  the  ISD  handbook  and  to  integrate  the  new 
procedures  with  those  currently  employed,  indoctrination  training  will  be  appropriate  for 
existing  and  newly  assigned  ISD  team  members.  Initially,  the  development  contractor  for 
the  handbook  will  conduct  the  training.  The  training  will  be  in  a  form  appropriate  for  in- 
house  conduct  necessary  to  indoctrinate  the  SMEs.  It  will  be  prepared  so  that  it  can  be 
integrated  into  the  existing  3306th  ISD  training  program. 

Section  3.  Training  Equipment  Acquisition  Process  Model 

This  section  provides  a  general  decision  process  for  acquiring  all  Air  Force  mainten¬ 
ance-training  equipment.  The  starting  point  for  this  acquisition  process  model  is  the 
functional  equipment-design  documents  produced  by  the  ISD  teams.  The  model  consists  of 
a  procedural/decision  sequence: 

1.  Validate  the  training  equipment  function  and  design  characteristics  documented 
as  a  result  of  ISD. 

2.  Determine  feasibility  of  the  validated  equipment  requirements  in  terms  of 
funding  estimates,  delivery  schedules,  and  engineering  state-of-the-art. 

3.  Justify  the  concept  and  cost  to  the  Program  Director. 

4.  Prepare  statement  of  work  (SOW)  and  request  for  proposal  (RFP)  to  ensure  the 
contractor  can  produce  adequate  training  equipment. 

5.  Select  contractual  source  by  assessing  proposals  for  technical  approach,  based  on 
comprehension  of  goals,  timely  delivery  schedule,  experience,  facilities,  personnel 
resources,  and  cost. 

6.  Reevaluate  and  complete  details  of  the  procurement  specification  to  assure 
concurrence  with  every  specific  requirement,  applying  special  attention  in  stating  to  the 
contractor  that  rigorous  test,  acceptance,  and  checkout  procedures  contained  in  the 
specification  will  be  strictly  enforced. 


7.  Closely  monitor,  within  contractual  legal  bounds,  the  development  and  produc¬ 
tion  process  to  assure  equipment  quality  and  timeliness  of  delivery. 

S.  Supervise  and  participate  in  the  specified  test,  acceptance,  and  checkout 
activities. 

Figure  3  shows  the  procedure/decision  sequence. 

Section  6.  Implications  for  SPO  Acquisition  Activities 

An  outgrowth  from  Procedural  Requirements  (AF)  will  be  the  development  of  an  SPO 
Training  Equipment  Acquisition  Manager's  Handbook  which  will  provide  both  guidance  and 
general  information.  The  handbook  is  intended,  primarily,  for  individual  SPO  acquisition 
managers,  but  will  also  serve  while  gearing  up  for  the  procurement.  The  handbook  will  be 
useful  for  training  new  SPO  personnel,  and  will  support  (1)  procedural  guidance,  (2)  use  of 
available  resources,  and  (3)  preparation  of  procurement  specifications. 

Procedural  Guidance.  A  major  problem  for  the  SPO  Training  Equipment  Acquisition 
Manager  is  knowing  the  steps  to  accomplish  major  activities  in  the  acquisition  process. 
While  the  handbook  cannot  comprehensively  deal  with  all  the  acquisition  procedures,  it 
will  focus  on  key  requirements  that  impact  individual  SPO  acquisition  managers.  The 
following  are  important  handbook  subjects: 

1.  Procedures  to  establish  and  maintain  continuing  coordination  with  the  originators 
of  maintenance-simulator  requirements. 

2.  Procedures  and  an  illustrative  model  for  setting  and  updating  acquisition 
program  milestones. 

3.  Procedures  and  principles  to  guide  the  acquisition  manager's  interactions  with 
contracting  officers. 

4.  Standardization  of  data  requirements. 

Utilization  of  Available  Resources.  Few  SPO  Training  Acquisition  Managers  are 
familiar  with  all  of  the  sources  of  assistance  to  the  acquisition  process  that  exist  outside 
the  SPO  itself.  The  SPO  handbook  will  include  reference  resources  for: 

1.  Documenting  the  expertise  and  experimental  study  capabilities  that  can  be 
provided  by  Air  Force  laboratory  scientists  and  engineering  support  organizations. 

2.  Detailing  the  procedures  necessary  to  investigate  the  appropriateness  of  any 
particular  resource  and  to  arrange  for  specific  assistance. 

3.  Providing  a  comprehensive  guide  to  all  Air  Force  documentation  relevant  to 
training-equipment  acquisition,  including  Air  Force  Regulations,  Air  Force  pamphlets. 
MIL-SPECs,  and  MIL-STDs. 

Preparation  of  Procurement  Specifications.  The  major  responsibility  of  the  Training 
Equipment  Acquisition  Manager  is  to  translate  a  functional  description  of  a  maintenance 
simulator,  developed  from  the  1SD  process,  into  a  prime  item  procurement  specification 
that  can  become  contractually  binding  on  a  training-equipment  manufacturer.  The  degree 
of  design  discretion  permitted  the  contractor  is  the  major  factor  in  structuring  such  a 
specification.  Therefore,  it  is  required  that  the  1SD  team  provide  the  acquisition  manager 
with  the  essential  functional  characteristics  of  the  device. 
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Model  Specifications  (AF) 

The  Model  Specifications  (AF)  document  provides  model  specifications  to  the  ISIS 
team  and  SPO  Training  Equipment  Acquisition  Manager.  These  model  specifications 
result  from  the  application  of  the  processes  detailed  in  Procedural  Requirements  (AF). 
and  will  structure  and  detail  the  descriptions  of  training  equipments  prepared  for  approval 
and  procurement. 

Two  model  specifications  are  presented: 

1.  Training  Equipment  Design  Requirements  for  (name  of  trainer  to  be  inserted). 


The  training/trainer  descriptions  developed  by  the  ISD  team  and  forwarded  to  the  SPO  for 
hardware/software  acquisition. 

2.  Training  Equipment  Functional  Specification  for  (name  of  trainer  to  be  inserted). 


The  formal  procurement  specification  that  become  part  of  the  Request  for  Proposal 
(RFP). 

An  appendix  provides  a  cross-reference  for  the  two  Air  Force  documents. 

DISCUSSION  AND  CONCLUSIONS 

The  task  of  developing  a  generic  specification  for  electromechanical  and  mechanical, 
organizational-level  Simulated  Avionics  Maintenance  Trainers  (SAMTs)  is  difficult  and 
complex.  This  is  evident  from  the  comparative  analysis  of  alternative  approaches 
provided  in  the  following  paragraphs. 

Evaluation  of  the  Navy  Process 


Although  it  appeared  that  the  Navy  study  offered  two  approaches  to  the  development 
of  maintenance-training  simulators,  the  two  approaches  can  be  considered  as  one,  since 
the  second  is  simply  a  derivative  of  the  first.  Various  aspects  of  the  process  are  discussed 
below. 

1.  Assumptions.  The  following  assumptions,  which  were  made  during  the  develop¬ 
ment  of  the  Navy  process,  define  the  maintenance  environment  and  functions  of 
maintenance  personnel. 

a.  Task  analysis  and  simulator  configuration  are  the  responsibilities  of  a 
specifications  contractor. 

b.  The  government  monitors  contractor  performance  of  task  analysis  and 
simulator  configuration  determination  through  periodic  reviews.  Reviews  will  be  used  to 
update  the  contractor's  previous  work,  to  evaluate  and  approve  contractor  decision 
criteria,  and  to  provide  overall  project  direction  at  key  decision  points. 

c.  The  government  project  director  will  ensure  liaison  between  contractor  and 
Navy  personnel  for  data  acquisition. 


d.  The  project  director  will  provide  an  SOW  and  RFP,  will  monitor  contractor 
performance,  and  will  accept  the  final  product. 

2.  Meeting  Goals  and  Objectives.  The  established  goals  and  objectives  of  the  Navy 
study  and,  therefore,  the  Navy  process,  were: 

a.  To  produce  an  actual  procurement  specification  for  a  maintenance-training 
simulator  to  be  used  at  a  NAMTRADET  in  electronics  organizational-level  maintenance  of 
the  A-6E  TRAM/DRS. 

b.  To  note  the  practical  concerns  of  the  specifications  contractor  and  the 
government  program  manager,  and  to  improve  the  development  process  by  specifying  for 
future  program  managers  a  generic  development  process,  information  requirements, 
possible  information  sources,  review  points,  and  critical  features  of  program  management. 

No  actual  procurement  specification  is  included  in  any  documents  in  the  Navy  Study. 
The  Phase  II  Document  (N)  does,  however,  contain  a  sample  procurement  specification 
that  can  be  used  as  the  SOW'  for  an  RFP. 

The  primary  objective  of  the  Navy  Study  was  to  provide  an  example  of  a  simulator 
system  specification.  Practical  concerns  of  the  government  project  director  were 
presented.  Information  sources  and  data  retrieval  procedures  were  presented  and  forms 
for  their  acquisition  were  suggested.  Guidance  to  aid  the  government  project  director  in 
his  review  of  the  specifications  contractor  performance  was  presented  and  review 
procedures  were  discussed  throughout  the  Navy  study  documents.  Critical  features  were 
explained  and  possible  problems  commented  on,  although  more  text  should  have  been 
devoted  to  this  subject. 

3.  Process  Application.  There  is  no  indication  that  the  Navy  process,  as  presently 
described  by  the  Phase  I  and  Phase  II  Documents  (N),  has  been  applied  to  the  actual 
procurement  of  a  maintenance-training  simulator.  It  is  implied  that  the  Navy  process  has 
some  development  experience  in  performing  task  analysis  and  simulator-configuration 
definition.  This  is  not  the  case  with  the  procurement  discussions  and  their  sample 
specification  forms. 

4.  Scope.  The  Navy  process  addresses  aspects  from  project  initiation  through 
acceptance  of  hardware.  It  does  not,  however,  address  installation,  operation,  and 
evaluation  of  the  simulator  in  the  training  environment. 

The  maintenance-training  simulator  development  process  defined  by  the  Navy  process 
does  not  include  a  feedback  function.  If  problems  are  met,  feedback  to  upgrade  problem 
areas  is  not  provided.  No  procedures  for  upgrading  the  Navy  process  are  specified  in  any 
of  the  documents. 

5.  Utility.  NAVPERSRANDCEN  is  tasked  with  the  development  of  generic 
specifications  for  both  electromechanical  and  mechanical,  organizational-level 
maintenance-training  simulators.  The  present  Navy  process  is  concerned  only  with 
organizational-level  electronic  maintenance-training  simulators.  Actual  reference  to 
electronics  maintenance  is  minimal,  however,  electronics  is  of  major  importance  only 
during  the  initial  identificiation  of  tasks  to  be  analyzed.  The  example  forms  to  be  used 
are  designed  for  electronic  devices,  and  the  discussions  of  government  project  director 
actions,  and  of  the  sample  SOW  for  the  specifications  contractor  are  in  terms  of 
electronic  maintenance.  However,  the  forms  can  be  easily  changed,  and  appropriate 
terms  included. 


The  sample  SOW  in  Phase  1  Document  (N)  provides  the  only  detailed  explanation  o( 
the  Navy  process,  although  a  flow  chart  (Appendix  B)  depicts  the  process  in  broad  terms. 
It  is  difficult  to  comprehend  fully  the  Navy  process  from  these  documents.  More 
explanation  is  needed  on  the  process  steps,  government  project-director  functions  and 
responsibilities,  the  specification  contractor's  actions,  and  in  particular,  justification  for 
the  order  and  action  of  the  individual  Navy  process  steps. 

Evaluation  of  the  Air  Force  Process 

1.  Assumptions.  The  following  assumptions,  which  were  made  by  the  developers  of 
the  Air  Force  process,  either  define  the  environment  in  which  the  process  is  to  be 
performed  and/or  specify  the  personnel  who  are  to  perform  the  individual  functions 
required  to  develop  maintenance-training  simulators. 

a.  The  approach  is  a  modified  Instructional  Systems  Development  (ISD) 

process. 

b.  Maintenance  training  instructors  from  the  Air  Force  will  serve  as  ISD 
members  (analysts). 

c.  A  separate  Air  Force  office,  the  Systems  Program  Office  (SPO),  will  provide 
all  procurement  and  acceptance  functions.  This  SPO  is  not  part  of  the  ISD  team. 

d.  Information  on  the  system  under  study  will  be  obtained  from  (1)  contractors 
who  designed  and/or  built  the  system,  (2)  existing  maintenance  training  on  that  system, 
and  (3)  maintenance  personnel  who  currently  perform  maintenance  on  that  system. 

e.  The  Air  Force  process  will  be  used  to  develop  all  maintenance-training 
course  materials,  not  just  to  develop  a  maintenance-training  simulator. 

f.  Review  of  the  ISD  results  will  be  performed  by  an  Air  Force  organization 
external  to  the  ISD  team  and  SPO. 

2.  Meeting  Goals  and  Objectives.  The  Model  Specifications  (AF)  document 
specified  that  the  goals  of  the  Air  Force  study  (which  resulted  in  the  development  of  the 
Air  Force  process)  were  to  build  baseline  knowledge  about  techniques,  procedures,  and 
principles  necessary  for  broad  applications  of  simulation  in  maintenance  training. 

The  stated  goal  has  been  met  by  the  Air  Force  study.  The  detailed  explanation  of  the 
3306th  T&ES  ISD  process,  the  SPO  Training  Equipment  Acquisition  Manager's  functions, 
the  problem  areas  found,  and  solutions  determined  all  point  toward  an  extensive 
development  of  baseline  data.  The  resulting  Air  Force  process  itself  is  broad  and  appears 
to  cover  most,  if  not  all,  aspects  of  maintenance  training  both  in  course  materials  and  in 
simulator  functional  design  and  procurement. 

3.  Process  Application.  In  developing  task  analysis  and  determining  simulator 
functions,  an  existing  successful  maintenance-training  simulator  development  was 
selected  as  the  baseline  for  the  Air  Force  Process.  The  development  of  the  SPO 
procedures  used  existing  Air  Force  procedures  for  baseline  data.  In  short,  the  Air  Force 
Process  was  developed  from  existing  procedures  and  is  the  refinement  of  successful 
maintenance-training  simulator  developments. 

4.  Scope.  Procedural  Requirements  (AF)  covers  the  full  development  cycle  of  a 
maintenance-training  simulator  from  course  concept  to  final  acceptance  of  the  simulator. 


5.  Utility.  Although  the  Air  Force  process  was  developed  for  the  Air  Force 
environment  and  was  tailored  to  specific  Air  Force  activities,  many  of  the  conceptual 
problems  are  common  to  the  Navy  and  Air  Force,  and  the  concepts,  procedures,  and 
specifications  appear  to  be  applicable  to  Navy  use.  The  process  of  developing  training  and 
training-equipment  requirements  for  new  or  existing  systems  can  be  used  to  develop  any 
type  of  maintenance-trainer  simulator. 

Comparative  Analysis 

The  Navy  and  Air  Force  processes  provide  separate  approaches  that  might  be  useful 
to  NAVPERSRANDCEN  as  a  baseline  from  which  to  develop  a  generic  specification  for  an 
organizational-level,  electromechanical,  and  mechanical  maintenance-training  simulator. 
There  are  differences,  however,  in  concepts  and  procedural  steps  that  must  be  considered. 
The  following  discussions  compare  these  differences  and  assess  their  impact  on  the 
successful  development  of  a  maintenance-training  simulator. 

To  provide  a  foundation  for  this  comparative  analysis,  three  procedure  areas,  which 
are  common  to  the  Navy  and  Air  Force  processes,  and,  in  most  cases,  to  any  training- 
simulator  development,  will  be  discussed: 

1.  Determining  training  requirements. 

2.  Selecting  media  to  meet  training  requirements. 

3.  Producing  a  procurement  specification  for  training  equipment  acquisition. 

Figure  4  shows  the  flow  between  these  procedural  areas  within  the  Navy  and  Air 
Force  Processes.  The  inherent  difference  between  the  processes  extends  even  to  the 
interrelation  of  common  procedural  areas. 
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Determining  Training  Requirements 


All  of  the  three  areas  common  to  the  Navy  and  Air  Force  processes  are  extremely 
important  to  the  successful  design  of  a  maintenance-training  simulator.  The  determina¬ 
tion  of  training  requirements,  however,  is  the  most  influential  of  the  three.  Development 
Guidelines  (N)  states  that  a  survey  performed  in  1977  by  Federman  and  Siegel  showed  that 
current  simulators  used  by  the  military  contained  defects  in  at  least  five  separate  areas: 
media,  fidelity,  job  orientation,  human  engineering,  and  logistics.  In  each  case,  the 
simulator  defects  were  traced  back  to  the  use  of  a  faulty  task  analysis  or  to  no  task 
analysis  prior  to  functional  simulator  design.  The  survey  results  strongly  suggest  that  "a 
simulator  designed  and  built  without  a  comprehensive  task  analysis  has  little  chance  of 
being  an  effective  or  even  relevant  learning  aid." 

The  analysis  of  the  training-requirements-determination  process  of  the  Navy  and  Air 
Force  will  be  confined  to  the  following  areas: 

1.  Personnel  assigned  to  the  determination  process. 

2.  Integration  of  the  maintenance  course  and  simulator. 

3.  Instructor  acceptance  of  the  simulator. 

4.  Task  classification  weaknesses. 

These  areas  are  discussed  in  the  following  paragraphs. 

Personnel  Assigned  to  the  Determination  Process.  The  Air  Force  ISD  process 
provides  that  future  instructors  will  be  assigned  as  ISD  team  members.  The  operational 
philosophy  is  to  select  individuals  to  become  ISD  analysts  who  are  already  experienced  and 
capable  instructors  in  specific  subsystems  specialities.  The  ISD  team  analysts,  therefore, 
have  complete  knowledge  of  Air  Force  desires  and  user  needs  coupled  with  instructor 
experience. 

The  Navy  process  requires  the  government  project  director  to  make  initial  deter¬ 
mination  of  training  requirements  and  the  specifications  contractor  to  complete  the  list. 
These  individuals  may  or  may  not  understand  the  user's  needs  as  well  as  the  ISD  team 
analysts  do.  The  contract  personnel  must  also  interact  through  the  Contracting  Technical 
Representative  and  other  governmental  personnel  to  perform  their  tasks.  Communication 
and  data  access  difficulties  can  arise. 

Integration  of  Maintenance  Course  and  Simulator.  The  Air  Force  process  provides  for 
the  development  of  not  just  a  maintenance-training  simulator  but  also  all  maintenance¬ 
training  course  materials.  The  maintenance-training  simulator,  if  developed,  plays  a  part 
in  the  course  materials;  it  becomes  an  integral  part  of  instruction  to  be  given  by  the 
entire  maintenance-training  course.  A  Plan  of  instruction  developed  by  the  Air  Force 
Process  is  used  to  ensure  integration  is  achieved. 

Development  Guidelines  (N)  states:  "With  a  proper  task  analysis,  there  remains  a 
strong  risk  of  training-mission  failure  without  the  integrated  (parallel)  development  of 
instructional  materials,  including  the  training  devices."  A  further  amplification  of  this 
statement  is  made  by  Phase  I  Document  (N):  "Experience  with  maintenance-training 
simulators  in  the  field  suggest  .  .  .  (that)  simulator  design  and  development  were 
inadequately  integrated  with  course  curriculum.  Often,  the  simulator  was  simply  an 
added  aid  to  training." 

Even  with  the  above  stated  requirement  by  the  Navy  study  for  the  integration  of  the 
maintenance-training  simulator  into  course  materials,  little  is  done  within  the  Navy 
process  to  ensure  this  integration.  If  emphasis  were  to  be  placed  on  this  integration, 
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statements  to  this  effect  should  have  been  included  in  the  discussion  to  aid  the  project 
director,  and  also  within  the  sample  SOW  for  the  specifications  contractor.  The  sample 
SOW  for  the  specifications  contractor  requires  the  contractor  to  designate  simulated 
tasks  and  those  left  to  conventional  means.  No  consideration  is  given  to  these  other 
means,  however. 

Instructor  Acceptance  of  the  Simulator.  Full  acceptance  of  the  simulator  by  the 
instructors  is  mandatory  if  it  is  to  be  successful,  regardless  of  how  comprehensive  and 
cost-effective  its  design.  The  introduction  to  the  Phase  I  Document  (N)  indicates  that  the 
instructor  has  often  been  overlooked. 

The  Air  Force  process  assigns  maintenance  instructors  as  1SD  team  analysts,  and  the 
instructor's  desires,  needs,  and  individual  instructional  philosophies  become  an  integral 
part  of  the  development  process  itself.  In  the  Navy  process,  on  the  other  hand,  little 
attention  is  paid  to  the  maintenance  instructor  either  in  comments  directed  toward  the 
project  director  or  in  the  sample  SOW. 

Task  Classification  Weakness.  In  the  Air  Force  training  requirements  determination 
processes,  tasks  must  be  classified  by  some  "undetermined"  criteria,  and  a  maintenance- 
task  taxonomy  is  used  to  link  characteristics  of  maintenance  simulators  which  can 
effectively  support  training.  Neither  the  criteria  nor  the  taxonomy  are  defined.  The  Air 
Force  study  reports  that  this  taxonomy  will  be  developed  in  later  technical  documents. 

The  Navy  process  provides  reference  materials  (in  the  sample  SOW)  to  aid  the 
specifications  contractor  in  task  classification  and  in  linking  simulator  functional 
characteristics. 

Selecting  Media  to  Meet  Training  Requirement 

To  ensure  the  development  of  an  effective  maintenance-training  simulator,  extreme 
care  must  be  taken  when  selecting  media  to  meet  the  training  requirements.  Develop¬ 
ment  Guidelines  (N)  states:  "Incorrect  or  incomplete  performance  of  this  (procedural) 
step  will  produce  a  defective  simulator." 

In  media  selection  in  the  Navy  and  Air  Force  processes,  differences  occur  in  three 
major  areas: 

1.  Requirements  for  simulator  development. 

2.  Methods  for  selecting  media. 

3.  Feedback  capabilities. 

These  areas  are  discussed  in  the  following  paragraphs. 

Requirements  for  Simulator  Development.  Media  selection  must  ensure  cost- 
effective  selection  of  training  materials.  Procedural  Requirements  (AF),  when  discussing 
problem  areas  that  exist  with  the  3306tr  T&ES  ISD  process,  points  out  that  many  times 
decisions  to  develop  maintenance-training  simulators  may  have  been  made  without 
investigating  more  cost-effective  alternatives.  This  idea  is  further  amplified  by  the 
statement  from  Procedural  Requirements  (AF):  "Deciding  to  use  a  simulator  must  occur 
only  after  a  careful  determination  that  a  maintenance  simulator  represents  the  most  cost- 
effective  approach." 

The  Air  Force  process  provides  for  selecting  the  most  cost-effective  means  of 
meeting  training  requirements,  and  a  maintenance-training  simulator  is  not  mandatory 


unless  it  is  the  most  cost-effective  training  aid.  In  the  Navy  process,  when  discussing  the 
project  director,  no  statement  or  implication  is  made  on  the  selection  of  cost-effective 
training  materials.  With  reference  to  the  specifications  contractor,  at  the  end  of  the 
media  selection  procedural  step,  Phase  1  Document  (N)  states:  "(On  the  assumption  that 
some  tasks  will  be  suitable  for  training  on  a  simulator),  the  contractor  shall  classify  the 
tasks  into  those  to  be  trained  on  a  simulator  and  those  to  be  trained  by  other  means 
(lectures,  slides,  etc.)  at  the  NAMTRADET."  Since  more  contractual  work  is  designated 
to  be  performed  by  the  specifications  contractor  in  the  area  of  functional  simulator 
design,  it  can  be  assured  that  "some  tasks"  will  be  selected  for  training  on  a  maintenance- 
training  simulator. 

Methods  for  Selecting  Media.  The  Air  Force  process  includes  procedures  for 
selecting  media  in  the  handbooks  that  will  be  developed.  In  the  Navy  process,  the 
specifications  contractor  is  required  to  select  media  using  some  classification  process 
scheme  "already  described  in  the  literature  of  educational  psychology  .  .  .,"  according  to 
Phase  I  Document  (N). 

Feedback  Capabilities.  It  is  not  realistic  to  consider  media  selection  as  a  separate 
entity  in  the  Air  Force  process,  for  media  selection  in  the  ISD  process  is  part  of  an 
iterative  procedure  of  course-materials  development.  Feedback  is  available  to  clarify 
problem  areas  that  may  arise  through  media  integration  of  training  requirements.  The 
Navy  process  requires  that  media  selection  be  performed  after  training  requirements  have 
been  determined.  No  specific  method  of  feedback  exists. 

Producing  a  Procurement  Specification  for  Training-Equipment  Acquisition 

An  objective  of  both  the  Navy  and  Air  Force  studies  is  to  document  the  conversion  of 
maintenance-training  Simulator  characteristics  into  a  specification  for  a  production 
simulator. 

Contract  and  procurement  procedures  differ  greatly  among  all  government  agencies, 
and,  in  particular,  within  Department  of  Defense  activities.  Due  to  the  differences 
between  the  Navy  and  Air  Force  procurement  procedures,  the  comparison  of  the  sample 
specifications  developed  in  the  two  studies  will  be  limited  to  the  following  issues. 

1.  Sample  specification  format. 

2.  Reference  materials  for  specification  development. 

3.  Aids  to  government  procurement  personnel. 

These  issues  are  discussed  in  the  following  paragraphs. 

Sample  Specification  Format.  The  Air  Force  study  uses  a  development  process.  The 
ISD  team  prepares  a  Training  Equipment  Design  Requirements  specification,  which  is  then 
incorporated  into  a  Training  Equipment  Functional  Specification  by  the  SPO  Training 
Equipment  Acquisition  Manager.  (The  Training  Equipment  Functional  Specification  is 
used  for  RFP  development.)  This  procedure  allows  for  standardization  of  communications 
between  the  ISD  team  and  SPO  Training  Equipment  Acquisition  Manager.  The  procedure 
appears  to  be  time-consuming  but  required  because  the  ISD  team  and  SPO  Training 
Equipment  Acquisition  Manager  are  separated. 

The  Navy  sample  specification  is  in  itself  an  actual  RFP.  It  excludes  some 
"boilerplate"  common  to  all  RFPs.  It  does  not,  however,  provide  the  structure  for 
specifying  simulator  design  characteristics  that,  by  the  Air  Force,  is  accomplished  by 
Training  Equipment  Design  Requirements  and  Training  Equipment  Functional  Specifica¬ 
tions. 


Reference  Materials  for  Specification  Development.  An  implied  objective  for  both 
the  Navy  and  Air  Force  studies  was  identification  of  applicable  military  standards, 
military  specifications,  Air  Force  regulations  or  Navy  directives,  and  Department  of 
Defense  directives  for  simulator  design  and  construction. 

The  Air  Force  study  provides  a  table  relating  the  simulator  design  and  construction 
requirements  to  the  documents  defining  those  requirements.  Also  included  in  the  table 
are  other  (usually  personnel)  sources  of  requirements  information.  The  Navy  study 
provides  this  information  within  the  sample  specification,  which  incorporates  paragraphs 
that  fulfill  the  requirements  of  various  military  standards. 

A  factor  that  may  have  affected  the  Air  Force  sample  specification  is  the  objective 
of  the  Air  Force  study  to  provide  as  much  design  latitude  on  the  part  of  the 
hardware/software  manufacturer  and  yet  still  meet  the  desired  training  requirements  and 
characteristics  of  the  simulator.  The  Air  Force  study  stated  that  many  of  the  above 
mentioned  documents  could  place  undue  limitations  upon  simulator  design  and  construc¬ 
tion,  thereby  increasing  cost  and  in  some  cases  defeating  training  effectiveness. 

Aids  to  Government  Procurement  Personnel.  For  both  studies,  assistance  to  aid  the 
government  procurement  personnel  was  defined  as  an  objective  of  the  studies.  The  Air 
Force  effort  is  limited  in  such  guidance.  Effort  was  expended  in  the  Navy  study  to 
delineate  government  personnel  actions  throughout  the  maintenance-simulator  design 
process.  These  actions  included  advanced  planning  tips,  sample  cover  letter  for  the  RFP, 
sample  CDRLs,  sample  software  specifications,  and  even  a  checklist  for  drafting  military 
specifications.  Even  though  some  of  the  aids  from  the  Navy  Study  are  elementary,  the 
information  is  useful  and  desirable. 


RECOMMENDATIONS 

Recommended  Training-Requirements  Determination 

It  is  recommended  that  Procedural  Requirements  (AF)  serve  as  the  reference  for  the 
development  of  training  requirements  and  media  selection  for  the  following  reasons: 

1.  The  existing  ISD  process  chosen  by  the  Air  Force  study  as  a  base  is  a  proved  and 
effective  process  for  maintenance-training  simulator  design  specifications.  The  Navy 
study,  on  the  other  hand,  uses  a  process  developed  from  limited  studies  and  is  not  based  on 
actual  simulator  design. 

2.  Procedural  Requirements  (AF)  appears  to  be  a  well  developed  and  easily 
understood  document.  Reasons  for  using  the  methods  chosen  are  clear  and  detailed. 
Criteria  to  be  met  by  the  planned  procedures  in  the  handbooks  are  clear  and  concise. 
Sources  cited  are  numerous  and  provide  a  wide  range  of  opinion  and  reference. 
Definitions  of  terms  are  included  and  their  use  is  consistent.  On  the  other  hand,  the 
Phase  I  Document  (N)  provides  little  background  information,  reasoning  behind  decisions, 
discussions  of  source  materials,  or  justification  for  the  process.  Many  of  the  discussions 
provide  limited  insight  on  the  development  of  maintenance-training  simulators.  Many 
definitions  appear  to  be  "non-government  standard,"  as  exemplified  by  the  term  project 
director  for  an  individual  who  appears  to  be  equivalent  to  project  engineer  in  standard 
government  terminology.  Consistency  in  terms  is  lacking. 

3.  Since  Procedural  Requirements  (AF)  provides  for  the  complete  development  of 
the  maintenance-training  materials,  a  simulator  designed  by  this  process  will  be  an 
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integral  part  of  the  training  course.  Though  this  problem  was  addressed  in  the 
introduction  to  Phase  I  Document  (N),  little  has  been  done  to  ensure  this  integration.  The 
Navy  simulator  will  therefore  become  an  "add-on"  to  the  course  and  must  be  integrated 
after  the  course  has  been  developed.  This  external  development  of  the  simulator  may  not 
exercise  the  skills  and  knowledge  required  by  course  materials. 

4.  The  development  of  a  maintenance-training  simulator  is  an  expensive  and  time- 
consuming  effort  by  the  government.  Procedural  Requirements  (AF)  does  not  impose  a 
mandatory  requirement  for  a  simulator  if  the  skills  and  knowledge  can  be  taught  by  other 
methods.  Phase  I  Document  (N)  provides  little  guidance  in  this  area. 

5.  Instructor  acceptance  is  a  key  factor  in  simulator  development  and  design  Fven 
an  "add-on"  simulator  can  provide  effective  training  if  the  instructor  recognises  its 
utility.  Procedural  Requirements  (AF)  requires  that  the  instructor  become  an  ISD  analyst 
during  development  of  the  training  simulator,  ensuring  instructor  acceptance.  Phase  1 
Document  (N)  avoids  instructor  inputs,  and  acceptance  of  the  simulator  by  the  instructor 
is  problematical. 

6.  Feedback  is  essential  in  any  development  process.  Feedback  provides  for 
correction  of  problems  as  they  occur.  Procedural  Requirements  (AF)  provides  for  an 
iterative  process  that  allows  feedback  during  training  requirements  and  media  selection. 
Phase  I  Document  (N)  provides  only  for  government  review  for  feedback.  This  limited 
feedback  is  not  likely  to  provide  the  scope,  depth,  and/or  timeliness  required  for  efficient 
specification  development. 

7.  It  has  become  increasingly  evident  in  past  years  that  contractors  fail  to  meet 
government  requirements.  These  failures  can  usually  be  traced  to  poor  communication. 
To  avoid  this  communication  problem,  Procedural  Requirements  (AF)  calls  for  Air  Force 
personnel  to  perform  the  simulator  specification  development.  Air  Force  personnel  must 
maintain  the  level  of  expertise  necessary  to  do  this  job.  On  the  other  hand,  Phase  I 
Document  (N)  places  development  responsibility  on  contractor  personnel.  Communication 
between  government  and  contractor  personnel  is  accomplished  only  through  reviews  at 
key  points,  and  because  these  reviews  are  limited  and  time  consuming  at  key  points,  it  is 
doubtful  that  they  provide  the  necessary  communication. 

The  following  problem  areas  will  arise  in  using  of  Procedural  Requirements  (AF)  as  a 
reference  when  developing  the  generic  specification  for  a  maintenance-training  simulator: 

1.  In  translating  Procedural  Requirements  (AF)  into  the  Navy  environment,  it  will 
be  difficult  to  integrate  the  contractor  into  the  Air  Force  process.  Because  of  the  limited 
Navy  in-house  manpower,  a  contractor  will  probably  be  used  to  determine  the  training 
requirements.  To  help  maintain  previously  defined  advantages  of  the  ISD  process, 
incorporation  of  instructor  inputs  and  detailed  government  reviews  must  be  achieved. 
This  is  possible  if  review  cycles  are  provided  frequently  and  existing  instructors  are 
utilized  as  major  input  data  sources.  Responsibility  for  developing  the  entire  course 
material  will  also  have  to  be  placed  on  the  contractor. 

2.  Communication  with  the  staff  of  the  Air  Force  Human  Resources  Laboratory  in 
February  1980  indicated  that,  the  ISD  handbook  was  near  completion.  This  ISD  handbook 
could  provide  many  of  the  procedures  needed  by  NAVPERSR ANDCEN  in  their  develop¬ 
ment  of  generic  specifications.  Its  utility  will  have  to  be  established. 

3.  The  Air  Force  Human  Resources  Laboratory  has  concluded  that  the  taxonomy 
criteria  needed  for  task  classification  is  more  difficult  to  develop  than  was  initially 
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thought.  They  have  therefore  excluded  this  taxonomy  for  the  ISD  handbook,  and  have, 
instead,  implemented  a  training  requirements  functional  characteristics  procedure.  This 
procedure  is  presented  as  a  form  that,  when  filled  in  and  reviewed  by  a  media  selection 
expert,  will  provide  the  same  function  as  the  taxonomy  criteria.  The  new  training 
requirements  functional  characteristics  procedure  should  be  evaluated,  and  its  appropri¬ 
ateness  to  NAVPERSRANDCEN  ascertained. 

Recommended  Sample  Specification 

It  is  recommended  that  Phase  II  Document  (N)  be  used  as  a  sample  specification 
reference  for  the  following  reasons: 

1.  Information  provided  by  Phase  II  Document  (N)  on  the  entire  Navy  procurement 
process  is  extensive.  Detailed  information  on  government  requirements  for  simulator 
design,  procurement  practices,  sample  letters,  and  Request  for  Proposal  are  included. 
Model  Specifications  (AF)  provides  only  a  format  for  presenting  ISD-developed  data  for 
possible  SOW  inclusion.  Some  mention  of  the  procurement  process  executed  in  the  Air 
Force  is  contained  in  Procedural  Requirements  (AF),  but  only  as  a  general  overview. 
Also,  the  Air  Force  and  Navy  procurement  practices  differ  so  greatly  that  a  detailed 
discussion  of  Air  Force  procurement  is  of  no  use  to  the  Navy. 

2.  The  Phase  II  Document  (N)  provides  many  aids  for  use  by  the  government  for 
simulator  procurement.  These  aids  are  well  written  and  provide  much  information 
unknown  to  some  government  personnel.  These  aids  are  written  in  a  form  understandable 
by  even  very  inexperienced  procurement  personnel  in  the  Navy. 

3.  The  results  of  the  specification  process  in  both  the  Navy  and  Air  Force  studies  is 
the  development  of  an  RFP  for  simulator  procurement  from  a  hardware/software 
contractor.  The  Model  Specifications  (AF)  document  (and  the  discussions  provided  in 
Procedural  Requirements  (AF))  provide  little,  if  any,  help  to  government  procurement 
personnel  in  writing  an  RFP.  The  Phase  II  Document  (N)  provides  significant  information 
in  RFP  development,  with  examples  and  sample  sections  for  maintenance-training 
simulator  RFPs. 

The  following  difficulties  may  be  met  in  using  Phase  II  Document  (N): 

1.  The  Phase  II  Document  (N)  format  is  not  developed  for  use  by  the  ISD  process  in 
Procedural  Requirements  (AF).  Some  method  for  incorporating  the  ISD  process  output 
data  into  a  form  usable  by  the  Phase  II  Document  (N)  needs  development.  A  possible 
solution  is  to  incorporate  into  the  Phase  II  Document  (N),  those  sections  of  model 
specifications  used  for  the  conversion  of  ISD  process  output  data  into  a  SOW. 

2.  An  objective  in  the  Air  Force  study  for  development  of  Model  Specifications 
(AF)  was  to  provide  the  hardware/software  contractor  as  much  latitude  in  simulator 
hardware/software  design  as  possible,  yet  provide  for  the  achievement  of  all  simulator 
design-specification  requirements.  It  is  questionable  whether  all  the  requirements  placed 
on  electronic  equipment  design  from  MIL-SPECs,  MIL-STDs,  DoD  directives,  and  Navy 
guidelines  are  applicable  to  a  maintenance-training  simulator  to  be  used  only  in  a  training 
and  not  a  combat  environment.  These  requirements  usually  place  many  restraints  on  the 
hardware/software  contractor  that  add  appreciably  to  the  simulator  cost. 

Qualifications  and  Special  Problems 


Although  the  recommended  documents  will  be  very  helpful  in  the  development  of 
generic  specifications,  there  are  questions  that  must  be  answered  if  the  development  is 


to  be  successful.  These  questions  arise  from  the  complexity  of  the  task.  Not  all  questions 
can  be  identified  at  this  time;  some  will  be  raised  during  development.  The  questions 
noted  below  must  be  addressed  if  proper  development  is  to  proceed. 

1.  What  is  a  generic  specification?  The  term  generic  specification  has  been  used 
repeatedly  throughout  this  analysis,  in  Navy  Coordination  Papers,  and  in  communication 
within  and  without  the  Navy.  Yet,  even  with  this  extensive  use  of  the  term,  a  clear 
definition  is  not  available.  In  the  context  where  this  term  has  been  used  by 
N AVPERSR ANDCEN,  a  possible  definition  might  be  "a  device  or  procedure  to  provide 
specific  information  about  a  simulator  for  procurement  by  the  government."  This 
definition  does  not  necessarily  correspond  to  usage  by  others  (such  as  the  Air  Force). 
Standardization  of  terminology  is  needed. 

2.  What  form  will  a  generic  specification  take?  The  forms  that  appear  to  have 
been  taken  by  both  the  Navy  and  Air  Force  processes  are: 

a.  A  procedure  or  process  for  determining  training  requirements  and  for 
selecting  media  to  fulfill  those  requirements.  In  the  Navy  process,  this  was  displayed  in 
the  sample  statement  of  work  in  the  Phase  I  Document  (N).  In  the  Air  Force  process,  this 
will  be  fulfilled  by  the  upcoming  ISD  handbook. 

b.  A  "form"  by  which  simulator  specifics  can  be  identified  for  manufacturing 
the  simulator.  The  Phase  II  Document  (N)  provided  this  function  in  the  form  of  a  sample 
specification  that  becomes  a  part  of  the  government  RFP.  The  Air  Force  process  will 
provide  this  with  a  combination  of  Model  Specifications  (AF),  which  documents  the 
desired  simulator  characteristics,  and  the  to-be-developed  SOP  handbook. 

3.  Who  will  determine  training  requirements  and  select  media?  The  use  of  a 
specifications  contractor  could  cause  the  final  developed  generic  specification  to  be  far 
different  than  if  in-house  personnel  are  used.  A  specifications  contractor  will  require 
monitoring  and  periodic  review  by  the  government,  and  monitoring  should  be  scheduled  in 
the  general  specification.  A  SOW  detailing  the  specifications  contractor's  action  should 
be  available  or  developed  from  the  generic  specification.  This  SOW  will  be  used  for 
contractor  selection  and  to  guide  the  contractor's  efforts.  Criteria  for  contractor 
expertise  and  experience  should  be  included  to  ensure  correct  selection  of  contractors. 

The  choice  of  in-house  user  personnel  for  determining  training  requirements  and 
media  selection  may  require  different  procedures  than  if  a  contractor  is  used.  The  level 
of  knowledge  and  skill  of  personnel  assigned  to  the  specification  development  must  be 
carefully  considered.  Training  of  these  individuals  also  needs  to  be  considered.  Some 
review  should  be  present  from  sources  external  to  the  project  staff.  The  depth  and  source 
of  this  review  will  need  to  be  determined. 

This  problem  can  be  extended  to  consider  whether  a  generic  specification  could 
be  developed  that  is  independent  of  the  choice  of  specifications  contractor  or  in-house 
user  personnel. 

4.  Is  a  universal  generic  specification  possible?  The  eventual  goal  of  the  Navy 
appears  to  be  a  general  maintenance  training  simulator  generic  specification.  The  Air 
Force  appears  to  have  set  this  goal  also,  and  may  very  well  have  achieved  it.  If  it  is 
concluded  that  this  type  of  generic  specification  can  be  developed,  the  next  question 
would  be  whether  that  generic  specification  could  be  adopted  for  all  simulator 
development,  and  not  just  for  maintenance  training. 
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Although  the  initial  cost  of  developing  a  maintenance  training  simulator  "generic 
specification,"  in  terms  of  both  money  and  manpower,  will  be  great,  the  potential 
benefits  also  appear  to  be  great.  It  is  important  that  the  generic  specification  be 
developed,  tested,  and  then  refined.  Several  iterations  may  be  required.  Using  this 
approach,  with  the  recommendations  of  this  report  as  a  starting  point,  it  is  likely  that  the 
potential  benefits  will  be  realized. 
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GLOSSARY  OF  KEY  TERMS 


Analysis 


Environment 


Generic  Specification 


Intermediate-Level  (I-Level) 
Maintenance 


This  report--a  comparative  analysis  of  the  Navy  and 
Air  Force  processes  or  the  development  of  main¬ 
tenance  training  simulators. 

Factors  that  affect  the  execution  of  the  main¬ 
tenance  training  simulator  development  process. 

A  device  or  procedure  that  leads  from  conception  of 
a  project,  through  development  of  a  maintenance 
training  simulator,  to  the  actual  procurement  of 
that  simulator. 

Maintenance  tasks  for  a  system/subsystem  perf¬ 
ormed  by  the  Intermediate  Maintenance  Activity 
(IMA).  The  IMA  is  separate  from  the  user  organiza¬ 
tion. 


Maintenance  Training  Simulator 


Maintenance  Task 


Media 


A  trainer  that  provides  hands-on  practice  for 
aspects  of  maintenance  jobs  selected  for  their 
criticality  and  learning  difficulty,  and  reproduced  to 
the  necessary  degree  of  fidelity  by  computer-gen¬ 
erated  events  and/or  indications. 

Action  by  the  maintenance  technician  to  keep 
equipment  in  operation  (e.g.,  preventive  mainte¬ 
nance  to  preclude  trouble;  troubleshooting  to  detect 
cause  of  malfunction;  repair  or  replacement  of  worn 
or  malfunctioning  parts;  and  calibration,  scheduled 
or  as  required). 

Training  aids  (e.g.,  slides,  text,  lectures)  supporting 
maintenance  instruction. 


Organizational-level  (O-Level) 
Maintenance 


Maintenance  tasks  performed  by  the  users  of  a 
system/subsystem. 


Procedural  Step 


A  specific  action  to  be  performed  within  a  pro¬ 
cedure. 


Procedures 


Step-by-step  instructions  on  how  to  perform  a  task. 


Process 


A  set  of  procedures. 


Procurement  Specification 


Project  Director  (Government 
Project  Director) 


A  document  that  defines  a  maintenance-training 
simulator  in  a  form  acceptable  to  the  government 
and  a  hardware/software  manufacturer  so  that  the 
simulator  may  be  manufactured. 

Synonymous  with  the  term  "government  project 
engineer/program  manager"  used  in  the  Phase  I 
Document  (N). 
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Specification  Contractor 


Study 

Task  Analysis 
Training  Requirements 


A  contractor  tasked  by  the  government  to  develop  a 
procurement  specification  for  a  maintenance  train¬ 
ing  simulator.  The  task  usually  begins  with  task 
analysis  and  results  in  a  procurement  specification. 

A  group  of  documents  developed  for  either  the  Navy 
or  Air  Force  that  detail  a  maintenance  training 
simulator  development  process. 

An  analysis  of  a  process  to  identify  maintenance 
training  tasks  and  to  determine  their  importance. 

Aspects  of  maintenance  training  necessary  to  train 
maintenance  technicians  to  the  desired  level  of  skill 
and  knowledge. 


LIST  OF  ABBREVIATIONS  AND  ACRONYMS 


AF 

AFHRL 

ASA 

BO 

CDRL 

CNO 

DID 

DoD 

DTA 

FRAMP 

FTL 

CTA 

IMA 

ISD 

MIL-SPEC 

MIL-STD 

N 

NAMTRADET 

NAVAIRDEVCEN 
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NAVEDTRA 
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Request  for  Proposal 
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Statement  of  Work 
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Test  and  Evaluation  Squadron 

Task  Identification  Matrix 

Technical  Proposal  Requirements 


APPENDIX  A 

TABLES  OF  CONTENTS  FROM  NAVY  STUDY  DOCUMENTS 


development  ct'i defines  (N) 

TABLE  OF  CONTENTS 


Page  No. 

SECTION  1.  INTRODUCTION .  1-1 

SECTION  2.  TASK  IDENTIFICATION .  2-1 

Introduction  .  2-1 

PTIM  Development .  2-3 

PTIM  Validation .  2-7 

TIM  Revision/Review/Final  TIM .  2-9 

SECTION  3.  TASK  ANALYSIS .  3-1 

Introduction  .  3-1 

Gross  Task  Analysis .  3-1 

Task  List  Gross  Division/Review .  3-5 

Detailed  Task  Analysis  .  3-7 

Final  Task  List  Division/Review .  3-7 

SECTION  4.  FUNCTIONAL  SIMULATOR  DESIGN  .  4-1 

Introduction  .  4-1 

Skill  Practice  Requirements  and  Media  Decisions .  4-1 

Practice  Requirements/Methods/Media  -  Review  .  4-3 

Detailed  Simulator  Concepts .  4-0 

Draft  Specification .  4-9 

Draft  Specification  Review  .  4-12 

SECTION  5.  POST-SPECIFICATION  ACTIVITIES  .  5-1 

SECTION  6.  PRE-SIMULATOR  FUNCTIONAL  SPECIFICATION 

CONTRACT  AWARD .  6-1 

Introduction  .  6-1 

Information  Requirements  and  General  Principles .  6-1 

Suggestions  for  Inclusion  in  a  Statement  of  Work  for 

Development  of  a  Functional  Simulator  Specification.  .  6-2 

SECTION  7.  REFERENCES .  7-1 


PHASE  I  DOCUMENT  (N) 


TABLE  OF  CONTENTS 


INTRODUCTION 


Background  .  , 

Plan  to  Develop  Guidelines  .  .  .  . 
Sources  Contributing  to  Guidelines 


PHASE  I  TASK  ANALYSES  AND  SIMULATOR  CONFIGURATION  . 

Initiation  of  Project  . 

Responsibilities  and  Constraints  . 

Overview  of  Project  Director's  Tasks  . 

Information  to  be  Collected  by  Project  Director  before 

Letting  the  Contract  . 

Forms  to  Aid  Information  Collection  . 

Decisions  Based  Upon  Information  Collection  . 

Letting  the  Contract  . 

Introduction  to  the  Sample  Statement  of  Work  .... 


STATEMENT  OF  WORK 


1 . 0  BACKGROUND . 

2.0  IDENTIFICATION,  SELECTION,  AND  DESCRIPTION 

OF  TASKS  .....  . 

2.1  Task  Identification  . 

2.1.1  Preliminary  Task  Identification  Matrix  (PTIM) 

2. 1.1.1  Preliminary  Task  List  for  Organizational- 

Level  Maintenance  . 

2. 1.1. 2  Validation  of  PTLOLM . 

2. 1.1. 3  Task  Identification  Report  and  Conference 

Agenda  . 

2. 1.1. 4  Task  Identification  Review  Conference  .  .  . 

2. 1.1. 5  Task  Identification  Conference  Minutes  .  . 

2.2  Task  Analysis  . 

2.2.1  Gross  Task  Analysis  (GTA)  . 

2.2.2  Relative  Importance  of  Tasks  . 

2.2.3  Commonality  of  Tasks  . 

2.2.4  Selection  of  Tasks  for  Training  at  NAMTRADET 

2.2.5  Task  Analysis  Report  and  Conference  Agenda 

2.2.6  Task  Analysis  Review  Conference  . 

2. 2.6.1  Task  Analysis  Conference  Minutes  . 

2.2.7  Final  Task  List  (FTL)  for  a  NAMTRADET  .  .  . 

2.2.8  Detailed  Task  Analysis  (DTA)  . 

2.2.9  Behavioral  Objectives  (BO)  . 

2.2.10  Tasks,  Types  of  Learning,  Method/Media 

Selection . .  . 


TABLE  OK  CONTENTS  (Cont.) 


3.0 

3.1 

3.2 

3.2.1 

3.2.2 

3.3 

3.3.1 

3.3.2 

3.3.3 

3.3.4 


SIMULATOR  DESIGN  . 

Preliminary  Simulator  Designs  . 

Preliminary  Simulator  Designs  Report  and 

Conference  Agenda  . 

Preliminary  Simulator  Design  Review  Conference 
Preliminary  Simulator  Design  Review  Conference 

Minutes  . 

Final  Simulator  Design  . 

Script  for  Task  Performance  . 

Final  Simulator  Design  and  Script  and 

Conference  Agenda  .  .  . 

Simulator  Design  and  Script  Review  . 

Simulator  Design  and  Script  Review  Conference 
Minutes  . 


Contract  Data  Requirement  List 


Data  Item  Descriptions  . 

Miscellaneous  Advice  to  the  Project  Director 


References 


PHASE  II  DOCUMENT  (N) 
TABLE  OF  CONTENTS 


INTRODUCTION  . 

PROCUREMENT  PLANNING  . 

Formal  Procurement  Planning  . 

Informal  Procurement  Planning  . 

CONTACT  WITH  CONTRACTORS . 

Basic  Policy . 

Contact  with  Contractors  prior  to  Solicitation 

for  Proposals . . 

Contact  during  the  Solicitation  Period . 

Contact  after  the  Solicitation  Period  . 

Debriefing  of  Unsuccessful  Offerors  . 

Technical  Direction  of  Contractors  . . 

GENERAL  INFORMATION  ON  SPECIFICATIONS  . 

Checklist  for  specifications  . 

SAMPLE  SPECIFICATION . 

Introduction . 

1.  Scope  . 

2.  Applicable  Documents . 

3.  Requirements . 

3.1  Materials,  parts,  and  processes  .... 

3.2  Design . 

3.3  Reliability  . . 

3. A  Maintainability . 

3.5  Transportability . 

3.6  Performance  . 

3.7  Details  of  components  . 

3.8  Electromagnetic  interference  suppression 

3.9  Dimensions . 

3 . 10  Weight . (.  '. . 

3.11  Color  . 

3.12  Finish . 

3.13  Nameplates  or  product  markings . 

3.14  Government-furnished  property  . 

3.15  Workmanship  .  . 

Excerpts  from  MIL-STD-961  on  Section  3.0.  .  . 


4.  Quality  Assurance  Provisions .  V-44 

4.1  Responsibility  for  inspection .  V-44 

4.2  Classification  of  inspections .  V-44 

4.3  Inspection  conditions .  V-44 

4.4  In-process  inspections  .  V-44 

4.3  Quality  conformance  inspections.  .  .  .  •  .  •  V-44 

4.6  Extent  of  testing  and  test  conditions.  .  .  V-48 

4.7  Material  to  accompany  training  devices  .  •  V-50 

4.8  Reliability  and  maintainability  demon-  V-50 

strat ions . 

5.  Packaging .  V-54 

5.1  Preservation-packaging . .  .  V-54 

5.2  Packing .  V-54 

5.3  Marking  for  shipment . .  V-54 

5.4  Disassembly  for  shipment .  V-54 

5.5  Moment . . .  .  .  .  V-54 

5.6  Base .  V-54 

5.7  Hoisting  instructions  for  heavy  trainers  .  V-54 

6.  Notes .  V-58 

6.1  Intended  use .  V-58 

6.2  Definitions . ...  V-58 

6.3  Ordering  data .  V-60 

Appendix .  V-64 

VI.  PARTS  AND  SECTIONS  OF  THE  REQUEST  FOR  PROPOSALS  ....  VI- 1 

Standard  Form  33 .  VI-3 

Examples  of  Section  E  through  J  of  Part  II .  VI-5 

Data  Item  Descriptions  (DIDs) .  VI-23 

Standard  Form  1423 .  VI-24 

Listing  of  Suggested  DIDs .  VI-26 

Cover  Letter  for  Request  for  Technical  Proposals.  .  .  Vl-28 

VII.  TECHNICAL  PROPOSAL  REQUIREMENTS,  PROPOSAL  EVALUATION  Vll-1 

PLANS  AND  PROPOSAL  EVALUATION  REPORTS  . 

Enclosures 

Guidelines  and  procedures  for  the  preparation  of  Technical  Pro¬ 
posal  Requirements  (TPR)  and  Proposal  Evaluation  Plans  (PEP).  Naval 
Training  Equipment  Center,  ED-PEG-S040A ,  April  1978. 

Guidelines  and  procedures  for  the  preparation  of  Proposal  Evalu¬ 
ation  Rc?ports  (PER).  Naval  Training  Equipment  Center,  ED- PEG -S041A, 
April  1978. 

Military  Standard  961,  72  Sept.  1975,  Outline  of  Feins  and  In 
strurt ions  for  the  Preparation  of  Specifications  and  Associated  Pnru- 
men t  s . 


A-5 


Military  Standard  490,  30  Oct.  1968,  Specification  l’t.m  in-,. 

Military  Specification  81821,  8  Feb.  1974,  Trainers,  Maintenance, 
Equipment  and  Services,  General  Specifications  for. 

Military  Specification,  23991,  20  Feb.  1974,  Training  Devices, 
Military,  General  Specification  for. 

Data  Item  Descriptions. 

Computer  System  for  Simulator 

(a)  Statements  for  Specification 

(b)  Outline  of  Functional  Requirements  for  Microcomputer 
Applications 

(c)  Computer  Software 


A-6 


.  •*.  C. 


Figure  B-]  Navy  Study  Simulator  Development  Process 


Figure  B-]  Navy  Study  Simulator  Development  Process  (continued) 


Figure  B-l  Navy  Study  Simulator  Development  Process  (continued) 


Navy  Study  Simulator  Development  Process  (continued) 


Figure  B-l  Navy  Study  Simulator  Development  Process  (continued) 
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Figure  D-l*  (Continued). 
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